We had a few days between arriving at Natalia's graduation, so we spent some time touring around Fontainebleau and Paris. The train into Paris is pretty short, about 40 minutes. Between the commuter trains and the Paris metro, Thomas was in seventh heaven! Thomas has been excited to see the Eiffel tower for a long time now, and the real thing didn't disappoint! It was raining a little bit our first day in Paris (the 19th), but that didn't deter plenty of folks from coming out to see the tower as well! There was a bit of a line up to the elevator to get to the top of the tower, so it was pretty dark by the time we got up. Thomas loved running around, looking at all the city lights, and especially being blown around by the high winds buffeting the tower that night! Martin enjoyed it too, most especially undergoing an emergency diaper change at the top! The next day (the 20th) we started out the day by taking a short walk from the hotel to the Château de Fontainebleau One neat piece on display is a plate painted with Niagara Falls around the 1830's. The Falls look much shorter in the painting than they do now. I wonder if that's what they really looked like back then? The boys weren't very interested in the interior of the Château, but did like running around the grounds! Later that afternoon we boarded the train headed for Paris again. We discovered shortly after the train left the station that we'd left one of our backpacks at the train station. Desperate, we called Natalia, who was still in Fonty, to see if she could get in touch with the station staff to see if they could hold the back for us. Natalia called back to say there was no way to get in touch with the station staff directly, and that we'd have to check lost & found (wherever that was!) the next day. Little did we know... ... Natalia had already found our backpack at the station, and had taken it for some joyriding! We were reunited with our poor backpack at Notre Dame. We finished the evening with a great dinner at Le Relais de l'Entrecôte. The following day we spent relaxing at the Jardin du Luxembourg, followed by a quick visit to the Eiffel Tower in the fog, then finished the evening at a restaurant called L'enfance de Lard in Paris, which in Melissa's opinion was the culinary highlight of the trip! She ordered a delicious steak tartare. Thomas devoured his grandfather's ceviche-style fish appetizer because we told him it was sashimi :) The food here really was delicious! The owners let Thomas grab as many candies as he wanted from their giant candy dish afterwards. I think all French desserts have to be lit on fire at some point.

This past Christmas, Melissa and I went to France along with our two boys and Melissa's family to attend the graduation of Melissa's sister from INSEAD. And since we were going to be in Europe for Christmas anyway, we all decided to head to the Alps for some skiing after the graduation. It was quite the adventure! We left from Toronto in the evening of December 17th. First stop: Frankfurt! The boys did really well (slept most of the way!) on the long flight to Frankfurt. When we arrived in Frankfurt, we noticed the hallways were packed with empty cots. Only later we realized that this was because hundreds of flights were being cancelled due to snow, and that thousands of people were going to be stranded in Frankfurt. Thank God our flight to Paris wasn't cancelled, and only left a bit late. We did suffer our first casualty in Frankfurt however: Lufthansa lost Pam's luggage in Frankfurt and it didn't make it to Paris with us. As I write this, she still doesn't have it back. We waited a long time in Paris' Charles de Gaul airport waiting for Pam's luggage to arrive. After a few hours we finally gave up, and met up with our very patient cab driver to take us to Fontainebleau... ...zzzzzz... We woke up to the charming sight of Fontainebleau with snow gently falling. After a quick dinner at a local crêperie, we promptly passed out en-masse in the hotel.

On November 24th we landed some changes we think are a big improvement to how we've been doing nightly builds. We've started doing nightly builds on the same revision across platforms, and where possible, on revisions that we already know compile. In addition, all of the nightly builds will share the same buildid. We pick the revision to build by looking at the past 24 hours of builds, and finding the latest one that built successfully on all platforms. If there is no such revision, we'll build the latest revision on the branch and hope for the best. We also do some extra checking to make sure we don't build a revision that's older than the previous nightly. Prior to this, we would trigger the nightly builds at the same time every day, but which revision and buildid each platform got was completely dependent on what time the builds actually started. There were many instances of nightly builds having different revisions between the various platforms. These changes are a big deal because they mean that we have a much better chance of getting a working nightly build every day (or at least one that compiles!), and all the platforms will be based on the same revision, regardless of when they are run. No more guessing if today's Windows build includes the same fixes as the Mac build! If you're interested in the implementation, which includes some pretty fun buildbot hacking, the details are in bug 570814. Background discussion as to how to choose the revision to use can be found in this thread.

I've just spent 3 days trying to debug some new code in buildbot. The code in question is to implement a change to how we do nightly builds such that they use the same revision for all platforms. I was hitting a KeyError exception inside buildbot's util.loop code, specifically at a line where it is trying to delete a key from a dictionary. In simple form, the loop is doing this:

for k in d.keys():
    if condition:
        del d[k] # Raises KeyError....sometimes...

Tricky bit was, it didn't happen every time. I'd have to wait at least 3 minutes between attempts. So I added a bunch of debugging code:

print d

print d.keys()

for k in d.keys():
    print k
    if condition:
        try:
            del d[k] # Raises KeyError....sometimes...
        except KeyError:
            print k in d # sanity check 1
            print k in d.keys() # sanity check 2

Can you guess what the results of sanity checks 1 and 2 were? 'k in d' is False, but 'k in d.keys()' is True. whhhaaaaa? Much head scratching and hair pulling ensued. I tried many different variations of iterating through the loop, all with the same result. In the end, I posted a question on Stack Overflow. At the same time, Bear and Dustin were zeroing in on a solution. The crucial bit here is that the keys of d are (follow me here...) methods of instances of my new scheduler classes, which inherit from buildbot.util.ComparableMixin...which implements __cmp__ and __hash__. __cmp__ is used in the 'k in d.keys()' test, but __hash__ is used in the 'k in d' test. Some further digging revealed that my scheduler was modifying state that ComparableMixin.__hash__ was referring to, resulting in the scheduler instances not having stable hashes over time. Meanwhile, on stackoverflow, adw came up with an answer that confirmed what Dustin and Bear were saying, and katrielalex came up with a simple example to reproduce the problem. In the end, the fix was simple, just a few extra lines of python code. Too bad it took me 3 days to figure out!

One of the biggest pieces of infrastructure keeping us attached to Tinderbox is the issue of build logs. Tinderbox has (and continues to be) the canonical source of a build or test's output. A while ago we made some changes that should help pry us loose from this close dependency on Tinderbox. Build & test logs for all regular depend & try builds are now available on FTP. For example, today's nightly build log for windows is available at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-27-06-mozilla-central/mozilla-central-win32-nightly-build26.txt.gz. It's also in the same directory as the build itself! Test logs are available for depend builds, e.g. http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64/1288215864/mozilla-central_leopard_test-crashtest-build1283.txt.gz. Again, all the builds, logs and other files (test archives, symbols, etc.) are in the same directory. Logs for your try builds and test are also available, e.g. http://stage.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[email protected]/tryserver-macosx64/. Here too all the logs and binaries are in the same directory. In addition you should be getting links to those logs in your try server notification emails. There's room for improvement here, and a few places where logs aren't getting put in all the right places, but this is still a major step forward. Enjoy! Note to future readers that a some of the links to the logs above will be invalid a few weeks after this post as old directories are purged.

I got a new Thinkpad T510 at work to replace my aging MacBook Pro. I asked for a Thinkpad instead of another MacBook because I wanted hardware with better hardware support, in particular the trackpad. I got into the habit of bringing a USB mouse everywhere I went because the trackpad on the MacBook was so unreliable on linux. So when my new T510 arrived, I was pretty excited. And, except for one tiny problem (of the PEBKAC kind), transferring all my files from the old machine to the new one went flawlessly. Here's how I set up the new machine:

• Download image from sysresccd.org. Follow the instructions to make a bootable USB drive.
• Boot up computer off USB drive. Resize the existing NTFS partition to be really small. Add 2 new partitions in the new-free space: one for the boot partition for linux, and one to be encrypted and be formatted with lvm.
• Format boot partition as ext3. Setup encrypted partition with 'cryptsetup luksFormat /dev/sda6; cryptsetup luksOpen /dev/sda6 crypt_sda6'. Setup LVM with 'pvcreate /dev/mapper/crypt_sda6'. Setup two volumes, one for swap, and one for the root partition.
• Connect network cable between old laptop and new one. Configure local network.
• Copy files from old /boot to new /boot.
• Copy files from old / to new /. Here's where I messed up. My command was: 'rsync -aPxX 192.168.2.1:/ /target/'.
• Install grub.
• Reboot!

At this point the machine came up ok, but wasn't prompting to decrypt my root drive, and so I had to do some manual steps to get the root drive to mount initially. Fixing up /etc/crypttab and the initramfs solved this. However even after this I was having some problems. I couldn't connect to wireless networks with Network Manager. I couldn't run gnome-power-manager. Files in /var/lib/mysql were owned by ntp! Then I realized that my initial rsync had copied over files preserving the user/group names, not the uid/gid values. And since I wasn't booting off a Debian image, the id/name mappings were quite different. Re-running rsync with '--numeric-ids' got all the ownerships fixed up. After the next reboot things were working flawlessly. Now after a few weeks of using it, I'm enjoying it a lot more than my MacBook Pro. It boots up faster. It connects to wireless networks faster. It suspends/unsuspends faster. It's got real, live, page-up/page-down keys! The trackpad actually works!

I've just pushed poster 0.7.0 to the cheeseshop. Thanks again to everybody who sent in bug reports, and for letting me know how you're using poster! It's really great to hear from users. poster 0.7.0 fixes a few problems with 0.6.0, most notably:

• Added callback parameters to MutipartParam and multipart_encode so you can add progress indicators to your applications. Thanks to Ludvig Ericson for the suggestion.
• Fixed a bug where posting to a url that returned a 401 code would hang. Thanks to Patrick Guido and Andreas Loupasakis for the bug reports.
• MultipartParam.from_params will now accept MultipartParam instances as the values of a dict object passed in. The parameter name must match the key corresponding to the parameter in the dict. Thanks to Matthew King for the suggestion.
• poster now works under python2.7

poster 0.7.0 can be downloaded from the cheeseshop, or from my website. Documentation can be found at https://atlee.ca/software/poster/ I'm planning on looking at python 3 compatibility soon. Also, if anybody has suggestions on a reliable way to test the streaming http code, I'm open to suggestions! My current methods result in intermittent failures because of the test harness I suspect. poster's code is now available on bitbucket.

This Monday (October 25, 2010), we will be renaming the branches in our buildbotcustom hg repository. The current 'default' branch will be renamed to be called 'buildbot-0.7', and the current 'buildbot-0.8.0' branch will be renamed to 'default'. What does this mean for you? You need to explicitly specify which branch you want to update to after pulling. If you're currently on 'default', you should update to the 'buildbot-0.7' branch. If you're currently on 'buildbot-0.8.0', you should update to the 'default' branch. I'll be running these steps, for those who are interested in the gory details:

#!/bin/sh

set -e

test -d buildbotcustom-branchdance && rm -rf buildbotcustom-branchdance

hg clone http://hg.mozilla.org/build/buildbotcustom buildbotcustom-branchdance

cd buildbotcustom-branchdance



hg update default

echo 'This is the old default branch' > README

hg add README

hg commit --close-branch -m "Closing old default branch"

hg branch buildbot-0.7

echo 'This is the buildbot-0.7 branch' > README

hg commit -m 'Moving default to buildbot-0.7 branch'



hg update buildbot-0.8.0

echo 'This is the old buildbot-0.8.0 branch' > README

hg add README

hg commit --close-branch -m "Closing old buildbot-0.8.0 branch"

hg branch -f default

echo 'This is the default branch' > README

hg commit -m 'Moving buildbot-0.8.0 to the default branch'



echo "Out"

hg out



echo "Heads"

hg heads

Something prompted me to look at the size of our codebase here in RelEng, and how much it changes over time. This is the code that drives all the build, test and release automation for Firefox, project branches, and Try, as well as configuration management for the various build and test machines that we have. Here are some simple stats: 2,193 changesets across 5 repositories...that's about 6 changes a day on average. We grew from 43,294 lines of code last year to 73,549 lines of code as of today. That's 70% more code today than we had last year. We added 88,154 lines to our code base, and removed 51,957. I'm not sure what this means, but it seems like a pretty high rate of change!

Amazingly, one of the most popular links on this site is the quick tip, Getting free diskspace in python. One of the comments shows that this method doesn't work on Windows. Here's a version that does:

import win32file

def freespace(p):
    """
    Returns the number of free bytes on the drive that ``p`` is on
    """
    secsPerClus, bytesPerSec, nFreeClus, totClus = win32file.GetDiskFreeSpace(p)
    return secsPerClus * bytesPerSec * nFreeClus

The win32file module is part of the pywin32 extension module.