As of November 1st, when you push a change to mozilla-central, the following builds and tests get triggered:

Linux optimized build mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse Talos Talos nochrome Talos jss Talos dirty Talos tp4 Talos cold Linux debug build + leak tests mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse Linux optimized + refcounting build mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse

Windows optimized build mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse XP Talos XP Talos nochrome XP Talos jss XP Talos dirty XP Talos tp4 Vista Talos Vista Talos nochrome Vista Talos jss Vista Talos dirty Vista Talos tp4 Windows debug build + leak tests mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse Windows optimized + refcounting build mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse

Mac OSX optimized build mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse Leopard Talos Leopard Talos nochrome Leopard Talos jss Leopard Talos dirty Leopard Talos tp4 Leopard Talos cold Mac OSX debug build + leak tests mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse Mac OSX optimized + refcounting build mochitest 1/5 mochitest 2/5 mochitest 3/5 mochitest 4/5 mochitest 5/5 everythingelse

Linux 64-bit build Maemo Build mochitest chrome crashtest mochitest 1/4 mochitest 2/4 mochitest 3/4 mochitest 4/4 reftest xpcshell Talos Tdhtml Talos Tgfx Talos Tp3 Talos Tp4 Talos Tp4 nochrome Talos Ts Talos Tsspider Talos Tsvg Talos Twinopen Talos non-Tp1 Talos non-Tp2 WinCE build Windows Mobile build Linux Fennec Desktop build Windows Fennec Desktop build Mac OSX Fennec Desktop build

That's 111 distinct build and test jobs that get spread out across our build and tests pools. A total of 40 machine hours per checkin in our main build, test and talos pools is used, plus an additional 25 machine hours on the mobile devices!!! In addition, we also do certain types of jobs on a periodic basis:

• Nightly builds
• XULRunner builds
• Shark builds
• Code coverage runs
• L10n repacks for 72 locales and 7 platforms (Windows, Mac OSX, Linux, Windows Fennec, Mac OSX Fennec, Linux Fennec, Maemo); that's 504 individual repacks!

In the course of collecting the data for this post, I've been constantly amazed at the amount of stuff that we're doing, and the scale of the infrastructure! The list above is just for our mozilla-central branch, and I've most likely missed something. We do similar amounts of work for our other branches as well: Try, mozilla-1.9.2, mozilla-1.9.1, TraceMonkey, Electrolysis, and Places. Things have certainly changed a lot in the past year.

Continuing our RelEng Blogging Blitz, I'm going to be discussing how and when tests get triggered in our build automation systems. We've got two basic classes of tests right now: unit tests, and performance tests, a.k.a. Talos. The unit tests are run on the same pool of machines that the builds are done on, while the performance tests are run on a separate pool of around 100 Mac Minis. Both kinds of tests are triggered in similar ways. For refcounting ("unittest") builds, once the compile step is complete, the binaries are packaged up with make package, the tests are packaged up with make package-tests, the symbols are packaged up with make buildsymbols, and then the whole lot is uploaded to stage.mozilla.org using make upload. Once they're uploaded, we have valid URLs that refer to the builds, tests, and symbols. We then trigger the relevant unit test runs on that build. When a slave is assigned this test run, it then downloads the build, tests, and symbols from stage and starts running the tests. On mozilla-central, we've also recently started to run unittests on optimized and debug builds. We're hoping to bring this functionality to mozilla-1.9.2 once all the kinks are worked out. For regular optimized builds, in addition to unittests, we also trigger performance tests on the freshly minted build. OSX builds are currently tested on Tiger and Leopard for mozilla-1.9.1 and mozilla-1.9.2, and on Leopard only for mozilla-central and project branches. Windows builds are tested on XP and Vista, and Linux builds are tested on Ubuntu. In addition to having tests triggered automatically by builds, the Release Engineering Sheriff can re-run unittests or performance tests on request!

As part of our RelEng Blogging Blitz, I'll give a quick overview of when and how builds get triggered on our main build infrastructure. There are three ways builds can be triggered. The first, and most common way, is when a developer pushes his or her changes to hg.mozilla.org. Our systems check for new changes every minute or so, and put new changes into a queue. Once the tree has been quiet for 3 minutes (i.e. no changes for 3 minutes), a new build request is triggered with all queued changes. If there is a free slave available, then a new build starts immediately, otherwise the build request is put in a queue. The second way builds are triggered is via a nightly scheduler. We start triggering builds on branches at 3:02am pacific local time (some branches are triggered at 3:32am or 4:02 am). We run at 3:02am to avoid problems with daylight savings <-> standard time transitions. In the fall there are two 2:59am's when we go back to standard time, and in the spring transition there is no 2:59am. The start times are staggered to avoid slamming hg.mozilla.org, or other shared resources. The last way builds can be triggered is manually. The Release Engineering Sheriff can trigger builds on specific revisions, or rebuild past builds pretty easily, so if you need a build triggered, contact your friendly neighbourhood RelEng Sheriff!