|
|
Yesterday was a bit of an overwhelming day. After getting home at 1am after a long bus ride home, I was unwinding by catching up on some news and email. I came across these two links, both of which really lifted my mood.
The first, Grokking the Zen of the Vi Wu-Wei, talks about a programmer’s journey from emacs to BBEdit to vim. This post is a great read in and of itself, but what’s really worth it, is the link around the middle of the post to http://stackoverflow.com/questions/1218390/what-is-your-most-productive-shortcut-with-vim/1220118#1220118. This was truly a joy to read. Definitely the best answer I’ve ever seen on Stack Overflow, and quite possibly the best discussion of vi I’ve ever read. It taught me a lot, but I enjoyed reading it for more than that. It was almost like being on a little adventure, discovering all these little hidden secrets about the neighbourhood you’ve been living in for years. Like I said, it was 1am.
The second, The Pope, the judge, the paedophile priest and The New York Times, gave me some reassurance that things aren’t always as they seem as reported by the media. Regardless of how you feel about the Church or the Pope, it seems that journalistic integrity has fallen by the wayside here. From the article:
Fr Thomas Brundage, the former Archdiocese of Milwaukee Judicial Vicar who presided over the canonical criminal case of the Wisconsin child abuser Fr Lawrence Murphy, has broken his silence to give a devastating account of the scandal – and of the behaviour of The New York Times, which resurrected the story.
It looks as if the media were in such a hurry to to blame the Pope for this wretched business that not one news organisation contacted Fr Brundage. As a result, crucial details were unreported.
The entire article is worth a read.
Johnathan posted links to 3 scripts he finds useful. His sattap script looked handy, so I hacked it up for linux. Run it to do a screen capture, and upload the image to a website you have ssh access into. The link is printed out, and put into the clipboard.
Hope you find this useful!
#!/bin/sh
# sattap - Send a thing to a place
set -e
SCP_USER='catlee'
SCP_HOST='people.mozilla.org'
SCP_PATH='~/public_html/sattap/'
HTTP_URL="http://people.mozilla.org/~catlee/sattap/"
FILENAME=`date | md5sum | head -c 8`.png
FILEPATH=/tmp/$FILENAME
echo Capturing...
import $FILEPATH
echo Copying to $SCP_HOST
scp $FILEPATH ${SCP_USER}@${SCP_HOST}:$SCP_PATH
echo Deleting local copy
rm $FILEPATH
echo $HTTP_URL$FILENAME | xclip -selection clipboard
echo Your file should be at $HTTP_URL$FILENAME, which is also in your paste buffer
As of November 1st, when you push a change to mozilla-central, the following builds and tests get triggered:
That’s 111 distinct build and test jobs that get spread out across our build and tests pools. A total of 40 machine hours per checkin in our main build, test and talos pools is used, plus an additional 25 machine hours on the mobile devices!!!
In addition, we also do certain types of jobs on a periodic basis:
- Nightly builds
- XULRunner builds
- Shark builds
- Code coverage runs
- L10n repacks for 72 locales and 7 platforms (Windows, Mac OSX, Linux, Windows Fennec, Mac OSX Fennec, Linux Fennec, Maemo); that’s 504 individual repacks!
In the course of collecting the data for this post, I’ve been constantly amazed at the amount of stuff that we’re doing, and the scale of the infrastructure! The list above is just for our mozilla-central branch, and I’ve most likely missed something. We do similar amounts of work for our other branches as well: Try, mozilla-1.9.2, mozilla-1.9.1, TraceMonkey, Electrolysis, and Places. Things have certainly changed a lot in the past year.
Continuing our RelEng Blogging Blitz, I’m going to be discussing how and when tests get triggered in our build automation systems.
We’ve got two basic classes of tests right now: unit tests, and performance tests, a.k.a. Talos. The unit tests are run on the same pool of machines that the builds are done on, while the performance tests are run on a separate pool of around 100 Mac Minis. Both kinds of tests are triggered in similar ways.
For refcounting (“unittest”) builds, once the compile step is complete, the binaries are packaged up with make package, the tests are packaged up with make package-tests, the symbols are packaged up with make buildsymbols, and then the whole lot is uploaded to stage.mozilla.org using make upload. Once they’re uploaded, we have valid URLs that refer to the builds, tests, and symbols. We then trigger the relevant unit test runs on that build. When a slave is assigned this test run, it then downloads the build, tests, and symbols from stage and starts running the tests.
On mozilla-central, we’ve also recently started to run unittests on optimized and debug builds. We’re hoping to bring this functionality to mozilla-1.9.2 once all the kinks are worked out.
For regular optimized builds, in addition to unittests, we also trigger performance tests on the freshly minted build. OSX builds are currently tested on Tiger and Leopard for mozilla-1.9.1 and mozilla-1.9.2, and on Leopard only for mozilla-central and project branches. Windows builds are tested on XP and Vista, and Linux builds are tested on Ubuntu.
In addition to having tests triggered automatically by builds, the Release Engineering Sheriff can re-run unittests or performance tests on request!
As part of our RelEng Blogging Blitz, I’ll give a quick overview of when and how builds get triggered on our main build infrastructure.
There are three ways builds can be triggered.
The first, and most common way, is when a developer pushes his or her changes to hg.mozilla.org. Our systems check for new changes every minute or so, and put new changes into a queue. Once the tree has been quiet for 3 minutes (i.e. no changes for 3 minutes), a new build request is triggered with all queued changes. If there is a free slave available, then a new build starts immediately, otherwise the build request is put in a queue.
The second way builds are triggered is via a nightly scheduler. We start triggering builds on branches at 3:02am pacific local time (some branches are triggered at 3:32am or 4:02 am). We run at 3:02am to avoid problems with daylight savings <-> standard time transitions. In the fall there are two 2:59am’s when we go back to standard time, and in the spring transition there is no 2:59am. The start times are staggered to avoid slamming hg.mozilla.org, or other shared resources.
The last way builds can be triggered is manually. The Release Engineering Sheriff can trigger builds on specific revisions, or rebuild past builds pretty easily, so if you need a build triggered, contact your friendly neighbourhood RelEng Sheriff!
Folks from RelEng are going to be blogging about various bits of our build, test, and release automation infrastructure this week.
Stay tuned for more!
Several members of the Release Engineering team are going to be blogging next week about various bits of the build, test, and release automation infrastructure for Firefox.
If there’s something about our infrastructure you’ve always wondered about, give us a shout and we’ll do our best to explain it!
I just spent a few minutes upgrading my blog to wordpress 2.7. Looks like everything went smoothly! I did this upgrade with mercurial queues again. WordPress 2.7 is supposed to have better upgrade support built in, so I may not need mercurial for future upgrades.
Please let me know if you notice anything strange or missing since the upgrade.
Well, it seems like all the cool kids are tweetering these days, so to keep up with the times, I’ve signed up on the twitter.
You can follow my twittering here: http://twitter.com/chrisatlee
Since Mozilla has started using Mercurial for source control, I thought I shoud get some hands on experience with it.
My WordPress dashboard has been nagging me to upgrade to the latest version for quite a while now. I was running 2.5.1 up until today, which was released back in April. I’ve been putting off upgrading because it’s always such a pain if you follow the recommended instructions, and I inevitably end up forgetting to migrate some customization I made to the old version.
So, to kill two birds with one stone, I decided to try my hand at upgrading WordPress by using Mercurial to track my changes to the default install, as well as the changes between versions of WordPress.
Preparation:
First, start off with a copy of my blog’s code in a directory called ‘blog’.
Download WordPress 2.5.1 and 2.6.3 (the version I want to upgrade to).
Import initial WordPress code:
tar zxf wordpress-2.5.1.tar.gz # NB: unpacks into wordpress/
mv wordpress wordpress-2.5.1
cd wordpress-2.5.1
hg init
hg commit -A -m 'wordpress 2.5.1'
cd .. Apply my changes:
hg clone wordpress-2.5.1 wordpress-mine
cd wordpress-mine
hg qnew -m 'my blog' my-blog.patch
hg locate -0 | xargs -0 rm
cp -ar ../blog/* .
hg addremove
hg qrefresh
cd .. The ‘hg locate -0′ line removes all the files currently tracked by Mercurial. This is needed so that any files I deleted from my copy of WordPress also are deleted in my Mercurial repository.
The result of these two steps is that I have a repository that has the original WordPress source code as one revision, with my changes applied as a Mercurial Queue patch.
Now I need to tell Mercurial what’s changed between versions 2.5.1 and 2.6.3. To do this, I’ll make a copy (or clone) of the 2.5.1 repository, and then put all the 2.6.3 files into it. Again, I use ‘hg locate -0 | xargs -0 rm’ to delete all the files from the old version before copying the new files in. Mercurial is smart enough to notice if files haven’t changed, and the subsequent commit with the ‘-A’ flag will add any new files or delete any files that were removed between 2.5.1 and 2.6.3.
Upgrade the pristine 2.5.1 to 2.6.3:
hg clone wordpress-2.5.1 wordpress-2.6.3
tar zxf wordpress-2.6.3 # NB: Unpacks into wordpress/
cd wordpress-2.6.3
hg locate -0 | xargs -0 rm
cp -ar ../wordpress/* .
hg commit -A -m 'wordpress-2.6.3'
cd .. Now I need to perform the actual upgrade to my blog. First I save the state of the current modifications, then pull in the 2.5.1 -> 2.6.3 changes from the wordpress-2.6.3 repository. Then I reapply my changes to the new 2.6.3 code.
Pull in 2.6.3 to my blog:
cd wordpress-mine
hg qsave -e -c
hg pull ../wordpress-2.6.3
hg update -C
hg qpush -a -m Voilà ! A quick rsync to my website, and the upgrade is complete!
I have to admit, I don’t fully grok some of these Mercurial commands. It took a few tries to work out this series of steps, so there’s probably a better way of doing it. I’m pretty happy overall though; I managed a successful WordPress upgrade, and learned something about Mercurial in the process! The next upgrade should go much more smoothly now that I’ve figured things out a bit better.
|
