Hello,
I have been working for the past few weeks on getting tinderbox running
in a test-only mode, where it downloads the latest hourly build,
unpacks it and runs tests on that instead of building from source each
run.
The main reasons for this are:
* VMs are nice for manageability of build machines, but don't give
stable enough results for perf tests
* It may be desirable to e.g. have a slower CPU or less RAM for perf
tests
* we really want to reduce cycle times for build machines
We currently have one Linux (bl-bldlnx01) and one Windows XP
(bl-bldxp01) machine reporting to the MozillaTest tree on
tinderbox.mozilla.org
So far the tests running are: Tp, Tp2 (Darin's Tp test, see bug
342089), Ts, Tdhtml and Txul. the tinderbox-client-code side, I have
taken a two-pronged approach; just to get up and running, a
TTinderbox mode has been added to tinderbox (checked into CVS).
In parallel, I have been working on extracting the test code from
tinderbox itself, with the intention of having a very simple test
harness.
I expect to check this performance test harness in over the next few
days, when the new "testing" module is sorted out (thanks to davel for
that). After we have tests running in this harness, I would like to
remove all of the test-related code from the tinderbox client (it's
responsibly for roughly half the code in build-seamonkey-util.pl, and
should hopefully make that file easier to understand). The current
"TTinderbox" mode will be removed at that time as well (I also
offer this as an excuse as to why it's implemented so hackishly :) ).
In the near term, this will make it much easier for everyone to add and
maintain performance tests. Long-term, I think we'll need a better
harness, but an important step is to understand exactly what it is we
have now.
The main outstanding issue with the test-only-tinderbox idea is the
ability to correlate checkins to points on the graph, since the time
the test started is not the same as when the build started. I have a
tinderbox server patch checked in that displays the start time if the
"quickparse" HTTP param is defined (quickparse already exists, this
just adds the timestamp of the build start time as a new column, so the
test-only machine can fetch it). There is a bug holding up tinderbox
server changes (bug 334592), once that's cleared the test-only machines
will be able to send graph data in with the timestamp of the build
start.
Current thinking is that the test-only machines will publish to the
tinderbox server just like builds do, and you can correlate the builds
and tests by what time the build stops and what time the equivalent
test starts in the waterfall display. As mentioned above, the graph
times will be the build start time, so we can correlate points on the
graph to checkins. course doing this correlation visually will be
much easier to do with the new graph server that Vlad has put together!
Longer-term, we should think about better ways to display test data.
For example, SAF's tinderbox2 server provides a separate table, which
I think is a lot more useful for quickly grasping the current
performance situation:
Feedback warmly welcomed, please respond to the mozilla.dev.builds
newsgroup.
Thanks!
Rob Helmer
build/release, moco
dev-quality mailing list
dev-quality (AT) lists (DOT) mozilla.org