Hope you find this useful!

#!/bin/sh
# sattap - Send a thing to a place
set -e
 
SCP_USER='catlee'
SCP_HOST='people.mozilla.org'
SCP_PATH='~/public_html/sattap/'
 
HTTP_URL="http://people.mozilla.org/~catlee/sattap/"
 
FILENAME=`date | md5sum | head -c 8`.png
FILEPATH=/tmp/$FILENAME
 
echo Capturing...
import $FILEPATH
echo Copying to $SCP_HOST
scp $FILEPATH ${SCP_USER}@${SCP_HOST}:$SCP_PATH
echo Deleting local copy
rm $FILEPATH
 
echo $HTTP_URL$FILENAME | xclip -selection clipboard
echo Your file should be at $HTTP_URL$FILENAME, which is also in your paste buffer

That’s 111 distinct build and test jobs that get spread out across our build and tests pools. A total of 40 machine hours per checkin in our main build, test and talos pools is used, plus an additional 25 machine hours on the mobile devices!!!

In addition, we also do certain types of jobs on a periodic basis:

• Nightly builds
• XULRunner builds
• Shark builds
• Code coverage runs
• L10n repacks for 72 locales and 7 platforms (Windows, Mac OSX, Linux, Windows Fennec, Mac OSX Fennec, Linux Fennec, Maemo); that’s 504 individual repacks!

In the course of collecting the data for this post, I’ve been constantly amazed at the amount of stuff that we’re doing, and the scale of the infrastructure! The list above is just for our mozilla-central branch, and I’ve most likely missed something. We do similar amounts of work for our other branches as well: Try, mozilla-1.9.2, mozilla-1.9.1, TraceMonkey, Electrolysis, and Places. Things have certainly changed a lot in the past year.

We’ve got two basic classes of tests right now: unit tests, and performance tests, a.k.a. Talos. The unit tests are run on the same pool of machines that the builds are done on, while the performance tests are run on a separate pool of around 100 Mac Minis. Both kinds of tests are triggered in similar ways.

For refcounting (”unittest”) builds, once the compile step is complete, the binaries are packaged up with make package, the tests are packaged up with make package-tests, the symbols are packaged up with make buildsymbols, and then the whole lot is uploaded to stage.mozilla.org using make upload. Once they’re uploaded, we have valid URLs that refer to the builds, tests, and symbols. We then trigger the relevant unit test runs on that build. When a slave is assigned this test run, it then downloads the build, tests, and symbols from stage and starts running the tests.

On mozilla-central, we’ve also recently started to run unittests on optimized and debug builds. We’re hoping to bring this functionality to mozilla-1.9.2 once all the kinks are worked out.

For regular optimized builds, in addition to unittests, we also trigger performance tests on the freshly minted build. OSX builds are currently tested on Tiger and Leopard for mozilla-1.9.1 and mozilla-1.9.2, and on Leopard only for mozilla-central and project branches. Windows builds are tested on XP and Vista, and Linux builds are tested on Ubuntu.

In addition to having tests triggered automatically by builds, the Release Engineering Sheriff can re-run unittests or performance tests on request!

There are three ways builds can be triggered.

The first, and most common way, is when a developer pushes his or her changes to hg.mozilla.org. Our systems check for new changes every minute or so, and put new changes into a queue. Once the tree has been quiet for 3 minutes (i.e. no changes for 3 minutes), a new build request is triggered with all queued changes. If there is a free slave available, then a new build starts immediately, otherwise the build request is put in a queue.

The second way builds are triggered is via a nightly scheduler. We start triggering builds on branches at 3:02am pacific local time (some branches are triggered at 3:32am or 4:02 am). We run at 3:02am to avoid problems with daylight savings <-> standard time transitions. In the fall there are two 2:59am’s when we go back to standard time, and in the spring transition there is no 2:59am. The start times are staggered to avoid slamming hg.mozilla.org, or other shared resources.

The last way builds can be triggered is manually. The Release Engineering Sheriff can trigger builds on specific revisions, or rebuild past builds pretty easily, so if you need a build triggered, contact your friendly neighbourhood RelEng Sheriff!

Stay tuned for more!

If there’s something about our infrastructure you’ve always wondered about, give us a shout and we’ll do our best to explain it!

Our end goal is to run unittests on our optimized and debug builds. We would stop doing our current reference counting builds, since the debug builds also have reference counting enabled.

To get there requires a few intermediate steps:

• Turn off tests for the current build+tests unittest job. We’re hoping to do this soon. The tests run on the packaged builds would still be active at this point (first phase of bug 507540).
• Turn on tests for debug builds (bug 372581). There’s a bit of work left to do here to make sure that hung processes don’t kill the buildbot master with multi-gigabyte log files.
• Turn on tests for optimized builds (bug 486783). This would include nightly and release builds eventually as well, and will give us test coverage on our windows PGO builds.
• Turn off original reference counting builds completely, along with the tests done on these builds (second phase of bug 507540).

At some point we’re also going to be changing how we run mochitests, by splitting the tests up according to which directory they’re in, and then running each of the subset of tests in parallel on different machines (bug 452861).

Martin Joseph AtLee was born by caesarean section on Wednesday morning and weighed in at 7 pounds, 15 ounces. The surgery went well for both Melissa and Martin. Mel had quite a bit of blood loss later in the day on Wednesday, giving us a bit of a scare as dozens of nurses and doctors rushed into her room! She recovered nicely the next day though.

Martin had a bit of jaundice, so he got to spend Friday night in a tanning salon!

Thomas has been a wonderful big brother, always wanting to hold and kiss his baby brother.

For our most recent Firefox releases, we’ve started using some new signing infrastructure that I’ve been working on over the past few months. There are quite a few reasons why the new infrastructure is faster:

• Faster hardware. We’ve moved from an aging single core system to a new quad-core 2.5 GHz machine with 4 GB of RAM.
• Concurrent signing of locales. Since we’ve got more cores on the system, we should take advantage of them! The new signing scripts spawn off 4 child processes, each one grabs one locale at a time to process.
• In-process compression/decompression. Our complete.mar files use bz2 compression for every file in the archive. The old scripts would call the ‘bzip2′ and ‘bunzip2′ binaries to do compression and decompression. It’s significantly faster to do these operations in-process.
• Better caching of signed files. The biggest win came from the simple observation that after you sign a file, and re-compress it to include in a .mar file, you should be caching the compressed version to use later. The old scripts did cache signed files, but only the decompressed versions. So to sign the contents of our mar files, the contents would have to be completely unpacked and decompressed, then the cache was checked, files were signed or pulled from cache as necessary, and then re-compressed again.

  Now, we unpack the mar file, check the cache, and only decompress / sign / re-compress any files that need signing but aren’t in the cache. We don’t even bother decompressing files that don’t need signing, another difference from the original scripts.

  Big thanks to Nick Thomas for having the idea for this at the Mozilla All-Hands in April.

As a result of all of this, signing all our locales can be done in less than 15 minutes now! See bug 470146 for the gory details.

The main bottleneck at this point is the time it takes to transfer all of the files to the signing machine and back again. For a 3.5 release, there’s around 3.5 GB of data to transfer back and forth, which takes on the order of 10 minutes to pull down to the signing machine, and another few minutes to push back after signing is done.