Posts about mercurial

Exporting MQ patches

I've been trying to use Mercurial Queues to manage my work on different tasks in several repositories. I try to name all my patches with the name of the bug it's related to; so for my recent work on getting Talos not skipping builds, I would call my patch 'bug468731'.

I noticed that I was running this series of steps a lot:

cd ~/mozilla/buildbot-configs

hg qdiff > ~/patches/bug468731-buildbot-configs.patch

cd ~/mozilla/buildbotcustom

hg qdiff > ~/patches/bug468731-buildbotcustom.patch

...and then uploading the resulting patch files as attachments to the bug. There's a lot of repetition and extra mental work in those steps:

  • I have to type the bug number manually twice. This is annoying, and error-prone. I've made a typo on more than one occasion and then wasted a few minutes trying to track down where the file went.
  • I have to type the correct repository name for each patch. Again, I've managed to screw this up in the past. Often I have several terminals open, one for each repository, and I can get mixed up as to which repository I've currently got active.
  • mercurial already knows the bug number, since I've used it in the name of my patch.
  • mercurial already knows which repository I'm in.

I wrote the mercurial extension below to help with this. It will take the current patch name, and the basename of the current repository, and save a patch in ~/patches called [patch_name]-[repo_name].patch. It will also compare the current patch to any previous ones in the patches directory, and save a new file if the patches are different, or tell you that you've already saved this patch.

To enable this extension, save the code below somewhere like ~/.hgext/mkpatch.py, and then add "mkpatch = ~/.hgext/mkpatch.py" to your .hgrc's extensions section. Then you can run 'hg mkpatch' to automatically create a patch for you in your ~/patches directory!


import os, hashlib



from mercurial import commands, util

from hgext import mq



def mkpatch(ui, repo, *pats, **opts):
    """Saves the current patch to a file called -.patch
    in your patch directory (defaults to ~/patches)
    """
    repo_name = os.path.basename(ui.config('paths', 'default'))
    if opts.get('patchdir'):
        patch_dir = opts.get('patchdir')
        del opts['patchdir']
    else:
        patch_dir = os.path.expanduser(ui.config('mkpatch', 'patchdir', "~/patches"))

    ui.pushbuffer()
    mq.top(ui, repo)
    patch_name = ui.popbuffer().strip()

    if not os.path.exists(patch_dir):
        os.makedirs(patch_dir)
    elif not os.path.isdir(patch_dir):
        raise util.Abort("%s is not a directory" % patch_dir)

    ui.pushbuffer()
    mq.diff(ui, repo, *pats, **opts)
    patch_data = ui.popbuffer()
    patch_hash = hashlib.new('sha1', patch_data).digest()

    full_name = os.path.join(patch_dir, "%s-%s.patch" % (patch_name, repo_name))
    i = 0
    while os.path.exists(full_name):
        file_hash = hashlib.new('sha1', open(full_name).read()).digest()
        if file_hash == patch_hash:
            ui.status("Patch is identical to ", full_name, "; not saving")
            return
        full_name = os.path.join(patch_dir, "%s-%s.patch.%i" % (patch_name, repo_name, i))
        i += 1

    open(full_name, "w").write(patch_data)
    ui.status("Patch saved to ", full_name)


mkpatch_options = [
        ("", "patchdir", '', "patch directory"),
        ]
cmdtable = {
    "mkpatch": (mkpatch, mkpatch_options + mq.cmdtable['^qdiff'][1], "hg mkpatch [OPTION]... [FILE]...")
}

Clobbering the trees

Today we landed some changes that will give developers self-serve clobber ability on our Mozilla Central / Mozilla 1.9.1 / Tracemonkey infrastructure.

In our current infrastructure, we have a large pool of slave machines for each platform that each build all the various branches. This makes it nice and easy to spin up new project and release branches, and automatically distributes jobs across branches. However, it can sometimes be confusing when tracking down a build or test failure. Sometimes, a particular machine needs to have its build directory cleaned out; and sometimes all the machines for one branch or build type need to be cleaned up.

Until now, this could only be done by RelEng by accessing the build machines directly.

But now you can do it too!

If you've got a valid LDAP account, head on over to http://build.mozilla.org/clobberer. You'll see a giant table, with lots of checkboxes on it.

If you check a box next to one of the slaves on a particular branch / builder, then the next time that slave runs a build on that branch, it will first delete the entire build directory, and then do a fresh checkout, and continue on with the rest of the build.

Selecting a builder-level checkbox merely selects all the slaves for that builder, and similarly, selecting the branch-level checkbox selects all the slaves for all the builders in that branch.

In addition, if a slave has not been clobbered in a configurable time period (currently set to 1 week), it will clobber on the next run.

Slaves are added to the database as they report in to ask for their clobber data, so it could take a little while for all the slave / builder / branch combinations to show up.

See bug 432236 for more information.