Once deployed, these changes will give us the ability to give real time access to various information about our build queue: the list of jobs waiting to start, and which jobs are in progress. This should help other tools like Tinderboxpushlog show more accurate information. One limitation of the upstream work so far is that it only captures a very coarse level of detail about builds: start/end time, and result code is pretty much it. No further detail about the build is captured, like which slave it executed on, what properties it generated (which could include useful information like the URL to the generated binaries), etc.

We’ve also been exporting a json dump of our build status for many months now. It’s been useful for some analysis, but it also has limitations: the data is always at least 5 minutes old by the time you look, and in-progress builds are not represented at all.

We’re starting to look at ways of exporting all this detail in a way that’s useful to more people. You want to get notified when your try builds are done? You want to look at which test suites are taking the most time? You want to determine how our build times change over time? You want to find out what the last all-green revision was on trunk? We want to make this data available, so anybody can write these tools.

Just how big is that firehose?

I think we have one of the largest buildbot setups out there and we generate a non-trivial amount of data:

• 6-10 buildbot master processes generating updates, on different machines in 2 or 3 data centers
• around 130 jobs per hour composed of 4,773 individual steps total per hour. That works out to about 1.4 updates per second that are generated

How you can help

This is where you come in.

I can think of two main classes of interfaces we could set up: a query-type interface where you poll for information that you are interested in, and a notification system where you register a listener for certain types (or all!) events.

What would be the best way for us to make this data available to you? Some kind of REST API? A message or event brokering system? pubsubhubbub?

Is there some type of data or filtering that would be super helpful to you?

The first, Grokking the Zen of the Vi Wu-Wei, talks about a programmer’s journey from emacs to BBEdit to vim. This post is a great read in and of itself, but what’s really worth it, is the link around the middle of the post to http://stackoverflow.com/questions/1218390/what-is-your-most-productive-shortcut-with-vim/1220118#1220118. This was truly a joy to read. Definitely the best answer I’ve ever seen on Stack Overflow, and quite possibly the best discussion of vi I’ve ever read. It taught me a lot, but I enjoyed reading it for more than that. It was almost like being on a little adventure, discovering all these little hidden secrets about the neighbourhood you’ve been living in for years. Like I said, it was 1am.

The second, The Pope, the judge, the paedophile priest and The New York Times, gave me some reassurance that things aren’t always as they seem as reported by the media. Regardless of how you feel about the Church or the Pope, it seems that journalistic integrity has fallen by the wayside here. From the article:

Fr Thomas Brundage, the former Archdiocese of Milwaukee Judicial Vicar who presided over the canonical criminal case of the Wisconsin child abuser Fr Lawrence Murphy, has broken his silence to give a devastating account of the scandal – and of the behaviour of The New York Times, which resurrected the story.

It looks as if the media were in such a hurry to to blame the Pope for this wretched business that not one news organisation contacted Fr Brundage. As a result, crucial details were unreported.

The entire article is worth a read.

If there’s something about our infrastructure you’ve always wondered about, give us a shout and we’ll do our best to explain it!

Why is this so cool?

Weave lets you securely synchronize parts of your browser profile between different machines. All your bookmarks, AwesomeBar history, saved passwords can be synchronized between your laptop, desktop and mobile phone. Your data is always encrypted with a private key that only you have access to.

Combine this with intelligent form-filling, automatic detection of OpenID-enabled sites, and you’ve got what is essentially single sign-on onto all your websites from all your browsers.

Now you’ll be able to sign into Firefox, and Firefox will know how to sign into all your websites.

Keep up the great work Labs!

I noticed that I was running this series of steps a lot:
cd ~/mozilla/buildbot-configs
hg qdiff > ~/patches/bug468731-buildbot-configs.patch
cd ~/mozilla/buildbotcustom
hg qdiff > ~/patches/bug468731-buildbotcustom.patch

…and then uploading the resulting patch files as attachments to the bug. There’s a lot of repetition and extra mental work in those steps:

• I have to type the bug number manually twice. This is annoying, and error-prone. I’ve made a typo on more than one occasion and then wasted a few minutes trying to track down where the file went.
• I have to type the correct repository name for each patch. Again, I’ve managed to screw this up in the past. Often I have several terminals open, one for each repository, and I can get mixed up as to which repository I’ve currently got active.
• mercurial already knows the bug number, since I’ve used it in the name of my patch.
• mercurial already knows which repository I’m in.

I wrote the mercurial extension below to help with this. It will take the current patch name, and the basename of the current repository, and save a patch in ~/patches called [patch_name]-[repo_name].patch. It will also compare the current patch to any previous ones in the patches directory, and save a new file if the patches are different, or tell you that you’ve already saved this patch.

To enable this extension, save the code below somewhere like ~/.hgext/mkpatch.py, and then add “mkpatch = ~/.hgext/mkpatch.py” to your .hgrc’s extensions section. Then you can run ‘hg mkpatch’ to automatically create a patch for you in your ~/patches directory!

import os, hashlib
 
from mercurial import commands, util
from hgext import mq
 
def mkpatch(ui, repo, *pats, **opts):
    """Saves the current patch to a file called <patch_name>-<repo_name>.patch
    in your patch directory (defaults to ~/patches)
    """
    repo_name = os.path.basename(ui.config('paths', 'default'))
    if opts.get('patchdir'):
        patch_dir = opts.get('patchdir')
        del opts['patchdir']
    else:
        patch_dir = os.path.expanduser(ui.config('mkpatch', 'patchdir', "~/patches"))
 
    ui.pushbuffer()
    mq.top(ui, repo)
    patch_name = ui.popbuffer().strip()
 
    if not os.path.exists(patch_dir):
        os.makedirs(patch_dir)
    elif not os.path.isdir(patch_dir):
        raise util.Abort("%s is not a directory" % patch_dir)
 
    ui.pushbuffer()
    mq.diff(ui, repo, *pats, **opts)
    patch_data = ui.popbuffer()
    patch_hash = hashlib.new('sha1', patch_data).digest()
 
    full_name = os.path.join(patch_dir, "%s-%s.patch" % (patch_name, repo_name))
    i = 0
    while os.path.exists(full_name):
        file_hash = hashlib.new('sha1', open(full_name).read()).digest()
        if file_hash == patch_hash:
            ui.status("Patch is identical to ", full_name, "; not saving")
            return
        full_name = os.path.join(patch_dir, "%s-%s.patch.%i" % (patch_name, repo_name, i))
        i += 1
 
    open(full_name, "w").write(patch_data)
    ui.status("Patch saved to ", full_name)
 
mkpatch_options = [
        ("", "patchdir", '', "patch directory"),
        ]
cmdtable = {
    "mkpatch": (mkpatch, mkpatch_options + mq.cmdtable['^qdiff'][1], "hg mkpatch [OPTION]... [FILE]...")
}

It seems like most people automatically assume that condoms are an important part of the solution to combating AIDS. It makes sense on some level I suppose; we’re reminded constantly of the importance of having “safe sex”, and how using a condom is the responsible thing to do. And I’m sure that condoms do reduce the risk of HIV transmission for any one given sexual encounter. But what are the effects over time? If condoms have a 99% success rate, that’s still 1 out of 100 failures. I’m not going to bet my life on a 1% chance of failure. Something with a 1% chance of occurring in a single event, has a 63% chance of occurring at least once over 100 events. Let’s say the prevention of transmission rate is 99.9%; there’s still a 9.5% chance of transmission over 100 sexual encounters in this scenario.

Now, big giant disclaimer here, I don’t know what the accepted statistics are on the effectiveness of condoms in preventing HIV transmission, either in ideal circumstances, or in actual usage.

I do know that the chances of failure for something definitely add up quickly over time, and they add up fast.

So it shouldn’t be a surprise to hear that, “We have found no consistent associations between condom use and lower HIV-infection rates, which, 25 years into the pandemic, we should be seeing if this intervention was working.” The full article can be read over at the National Review Online.

In two places I know of that have had success in combating AIDS, Uganda and the Philippines, the primary focus was on having faithful, monogamous sexual practices. And it makes sense why this works. If people have fewer sexual partners, then the risk of transmission in the general population is reduced.

So maybe the Pope is right when he said, “If the soul is lacking, if Africans do not help one another, the scourge cannot be resolved by distributing condoms; quite the contrary, we risk worsening the problem. The solution can only come through a twofold commitment: firstly, the humanization of sexuality, in other words a spiritual and human renewal bringing a new way of behaving towards one another; and secondly, true friendship, above all with those who are suffering, a readiness – even through personal sacrifice – to be present with those who suffer. And these are the factors that help and bring visible progress.” (my emphasis)

I think he is.

Thanks to Mulier Fortis for the link to the National Review article.

Please let me know if you notice anything strange or missing since the upgrade.

You can follow my twittering here: http://twitter.com/chrisatlee

Documentation can also be found here.