Skip to main content

Posts about mozilla (old posts, page 1)

XULRunner Nightlies now available

As Mossop mentioned, I've been working for the past week on getting XULRunner nightly builds up and running.

I'm happy to announce that they're now available!

The first builds of XULRunner for Linux (i686 and x86_64), Windows and OS X (i386 and ppc) for both mozilla-1.9.1 and mozilla-central (1.9.2) are finishing up, and are available (or will be soon!) at http://ftp.mozilla.org/pub/mozilla.org/xulrunner/nightly/. Fresh builds will be available every night.

Enjoy!

Exporting MQ patches

I've been trying to use Mercurial Queues to manage my work on different tasks in several repositories. I try to name all my patches with the name of the bug it's related to; so for my recent work on getting Talos not skipping builds, I would call my patch 'bug468731'.

I noticed that I was running this series of steps a lot:

cd ~/mozilla/buildbot-configs

hg qdiff > ~/patches/bug468731-buildbot-configs.patch

cd ~/mozilla/buildbotcustom

hg qdiff > ~/patches/bug468731-buildbotcustom.patch

...and then uploading the resulting patch files as attachments to the bug. There's a lot of repetition and extra mental work in those steps:

  • I have to type the bug number manually twice. This is annoying, and error-prone. I've made a typo on more than one occasion and then wasted a few minutes trying to track down where the file went.
  • I have to type the correct repository name for each patch. Again, I've managed to screw this up in the past. Often I have several terminals open, one for each repository, and I can get mixed up as to which repository I've currently got active.
  • mercurial already knows the bug number, since I've used it in the name of my patch.
  • mercurial already knows which repository I'm in.

I wrote the mercurial extension below to help with this. It will take the current patch name, and the basename of the current repository, and save a patch in ~/patches called [patch_name]-[repo_name].patch. It will also compare the current patch to any previous ones in the patches directory, and save a new file if the patches are different, or tell you that you've already saved this patch.

To enable this extension, save the code below somewhere like ~/.hgext/mkpatch.py, and then add "mkpatch = ~/.hgext/mkpatch.py" to your .hgrc's extensions section. Then you can run 'hg mkpatch' to automatically create a patch for you in your ~/patches directory!

import os, hashlib



from mercurial import commands, util

from hgext import mq



def mkpatch(ui, repo, *pats, **opts):
    """Saves the current patch to a file called -.patch
    in your patch directory (defaults to ~/patches)
    """
    repo_name = os.path.basename(ui.config('paths', 'default'))
    if opts.get('patchdir'):
        patch_dir = opts.get('patchdir')
        del opts['patchdir']
    else:
        patch_dir = os.path.expanduser(ui.config('mkpatch', 'patchdir', "~/patches"))

    ui.pushbuffer()
    mq.top(ui, repo)
    patch_name = ui.popbuffer().strip()

    if not os.path.exists(patch_dir):
        os.makedirs(patch_dir)
    elif not os.path.isdir(patch_dir):
        raise util.Abort("%s is not a directory" % patch_dir)

    ui.pushbuffer()
    mq.diff(ui, repo, *pats, **opts)
    patch_data = ui.popbuffer()
    patch_hash = hashlib.new('sha1', patch_data).digest()

    full_name = os.path.join(patch_dir, "%s-%s.patch" % (patch_name, repo_name))
    i = 0
    while os.path.exists(full_name):
        file_hash = hashlib.new('sha1', open(full_name).read()).digest()
        if file_hash == patch_hash:
            ui.status("Patch is identical to ", full_name, "; not saving")
            return
        full_name = os.path.join(patch_dir, "%s-%s.patch.%i" % (patch_name, repo_name, i))
        i += 1

    open(full_name, "w").write(patch_data)
    ui.status("Patch saved to ", full_name)


mkpatch_options = [
        ("", "patchdir", '', "patch directory"),
        ]
cmdtable = {
    "mkpatch": (mkpatch, mkpatch_options + mq.cmdtable['^qdiff'][1], "hg mkpatch [OPTION]... [FILE]...")
}

Release Engineering Sheriffs

Picking up a thread that was being discussed last year about suggested changes to sheriffing...

Starting next week (on the 17th), there will be one person from release engineering designated to be the RelEng Sheriff for the week. This person will be responsible for various duties, most important of which will be to be available in #developers to help the developer sheriff track down issues with build machines, test failures, or other infrastructure problems.

For more information, and for the current schedule of RelEng Sheriffs, please see the ReleaseEngineering:Sheriffing wiki page.

Clobbering the trees

Today we landed some changes that will give developers self-serve clobber ability on our Mozilla Central / Mozilla 1.9.1 / Tracemonkey infrastructure.

In our current infrastructure, we have a large pool of slave machines for each platform that each build all the various branches. This makes it nice and easy to spin up new project and release branches, and automatically distributes jobs across branches. However, it can sometimes be confusing when tracking down a build or test failure. Sometimes, a particular machine needs to have its build directory cleaned out; and sometimes all the machines for one branch or build type need to be cleaned up.

Until now, this could only be done by RelEng by accessing the build machines directly.

But now you can do it too!

If you've got a valid LDAP account, head on over to http://build.mozilla.org/clobberer. You'll see a giant table, with lots of checkboxes on it.

If you check a box next to one of the slaves on a particular branch / builder, then the next time that slave runs a build on that branch, it will first delete the entire build directory, and then do a fresh checkout, and continue on with the rest of the build.

Selecting a builder-level checkbox merely selects all the slaves for that builder, and similarly, selecting the branch-level checkbox selects all the slaves for all the builders in that branch.

In addition, if a slave has not been clobbered in a configurable time period (currently set to 1 week), it will clobber on the next run.

Slaves are added to the database as they report in to ask for their clobber data, so it could take a little while for all the slave / builder / branch combinations to show up.

See bug 432236 for more information.

Automated Talos Analysis

As part of one of our goals in Release Engineering this quarter, I'm investigating whether we can automatically detect variance in Talos performance data. Automatically detecting these changes in performance results would be a great help to developers and tree sheriffs.

Imagine if the Tinderbox tree could be made to burn if a performance regression was detected?

There are lots of possibilities if we can get this working: regressions could cause the tree to burn, firebot could spam #developers with information, try-talos data could be compared more easily to the baseline data, or we could automatically back out changes that cause regressions! :P

This is also exciting, because it allows us to consider moving towards a pool-o'-slaves model for the Talos machines, just like we have for build and unittests right now. Having Talos use a pool-o'-slaves allows us to scale to additional project / release branches much more quickly, and allows us to be more flexible in allocating machines across branches.

I've spent some time over the past few weeks playing around with data from graph server, bugging Johnathan, and having fun with flot, and I think I've come up with a workable solution.

How it works

I grab all the data for a test/branch/platform combination, and merge it into a single data series, ordered by buildid (the closest thing we've got right now to being able to sort the data in the same order in which changes landed).

Individual data points are classified into one of four buckets:

  • "Good" data. We think these data points are within a certain tolerance of the expected value. Determining what the expected value is a bit tricky, so read on!
  • "Spikes". These data points are outside of the specified tolerance, but don't seem to be part of an ongoing problem (yet). Spikes can be caused by having the tolerance set too low, random machine voodoo, or not having enough data to make a definitive call as to if it's a code regression or machine problem.
  • "Regressions". When 3 or more data points are outside of the tolerance in the same direction, we assume this is due to a problem with the code, and flag it as a regression.
  • "Machine problem". When the last 2 data points from the same machine have been outside of the tolerance, then we assume this is due to a problem with the machine.

For the purposes of the algorithm (and this post!), a regression is a deviation from the expected value, regardless of it's a performance gain or loss.

At this point the tolerance criteria is being set semi-manually. For each test/branch/platform combination, the tolerance is set as a certain number of standard deviations. The expected value is then determined by going back in the performance data history and looking for a certain sized window of data where no point is more than the configured number of standard deviations from the average. This can change over time, so we re-calculate the expected value at each point in the graph.

Initial Results

As an example, here's how data from Linux Tp3 tests on the Mozilla 1.9.2 branch is categorized: Linux Tp3 Data for Mozilla 1.9.2

Or, if you have a canvas-enabled browser, check out this interactive graph.

A window size of 20 and a standard deviation threshold of 2.5 was used here for this data set.

The green line represents all the good data. The orange line (which is mostly hidden by the green line), represents the raw data from the 3 Linux machines running that test.

The orange circles represent spikes in the data, red circles represent regressions, and blue circles represent possible machine problems.

For the most part we can ignore the spikes. Too many spikes probably means we need to tighten our tolerance a bit

There are two periods of to take notice of on this graph:

  • Jan 12, around noon, a regression was detected. Two orange spike circles are followed by three red regression circles. Recall that we wait for the 3rd data point to confirm an actual regression.
  • Jan 30, around noon, a similar case. Two orange spike circles, followed by regression points.

Although in these cases, the regression was actually a win in terms of performance, it shows that the algorithm works. The second regression is due to Alice unthrottling the Talos boxes.

In both cases, a new expected value is found after the data levels off again.

The analysis also produces some textual output more suitable for e-mail, nagios or irc notification, e.g.:

Regression: Tp3 decrease from 417.974 to 235.778 (43.59%) on Fri Jan 30 11:34:00 2009. Linux 1.9.2 build 20090130083434

http://graphs.mozilla.org/#show=395125,395135,395166&sel=1233236074,1233408874

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=7f5292b5b9e2&tochange=f1493cf102b9

My code can be found on http://hg.mozilla.org/users/catlee_mozilla.com/talos-grokker.

Patches or comments welcome!

python reload: danger, here be dragons

At Mozilla, we use buildbot to coordinate performing builds, unit tests, performance tests, and l10n repacks across all of our build slaves.

There is a lot of activity on a project the size of Firefox, which means that the build slaves are kept pretty busy most of the time.

Unfortunately, like most software out there, our buildbot code has bugs in it. buildbot provides two ways of picking up new changes to code and configuration: 'buildbot restart' and 'buildbot reconfig'.

Restarting buildbot is the cleanest thing to do: it shuts down the existing buildbot process, and starts a new one once the original has shut down cleanly. The problem with restarting is that it interrupts any builds that are currently active.

The second option, 'reconfig', is usually a great way to pick up changes to buildbot code without interrupting existing builds. 'reconfig' is implemented by sending SIGHUP to the buildbot process, which triggers a python reload() of certain files.

This is where the problem starts.

Reloading a module basically re-initializes the module, including redefining any classes that are in the module...which is what you want, right? The whole reason you're reloading is to pick up changes to the code you have in the module!

So let's say you have a module, foo.py, with these classes:

class Foo(object): def foo(self): print "Foo.foo"

class Bar(Foo): def foo(self): print "Bar.foo" Foo.foo(self)

and you're using it like this:

>>> import foo

>>> b = foo.Bar()

>>> b.foo()

Bar.foo

Foo.foo

Looks good! Now, let's do a reload, which is what buildbot does on a 'reconfig':

>>> reload(foo)



>>> b.foo()

Bar.foo

Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/catlee/test/foo.py", line 13, in foo
    Foo.foo(self)
TypeError: unbound method foo() must be called with Foo instance as first argument (got Bar instance instead)

Whoops! What happened? The TypeError exception is complaining that Foo.foo must be called with an instance of Foo as the first argument. (NB: we're calling the unbound method on the class here, not a bound method on the instance, which is why we need to pass in 'self' as the first argument. This is typical when calling your parent class)

But wait! Isn't Bar a sub-class of Foo? And why did this work before? Let's try this again, but let's watch what happens to Foo and Bar this time, using the id() function:

>>> import foo

>>> b = foo.Bar()

>>> id(foo.Bar)

3217664

>>> reload(foo)



>>> id(foo.Bar)

3218592

(The id() function returns a unique identifier for objects in python; if two objects have the same id, then they refer to the same object)

The id's are different, which means that we get a new Bar class after we reload...I guess that makes sense. Take a look at our b object, which was created before the reload:

>>> b.__class__



>>> id(b.__class__)

3217664

So b is an instance of the old Bar class, not the new one. Let's look deeper:

>>> b.__class__.__bases__

(,)

>>> id(b.__class__.__bases__[0])

3216336

>>> id(foo.Foo)

3218128

A ha! The old Bar's base class (Foo) is different than what's currently defined in the module. After we reloaded the foo module, the Foo class was redefined, which is presumably what we want. The unfortunate side effect of this is that any references by name to the class 'Foo' will pick up the new Foo class, including code in methods of subclasses. There are probably other places where this has unexpected results, but for us, this is the biggest problem.

Reloading essentially breaks class inheritance for objects whose lifetime spans the reload. Using super() in the normal way doesn't even work, since you usually refer to your instance's class by name:

class Bar(Foo):
    def foo(self):
        print "Bar.foo"
        super(Bar, self).foo()

If you're using new-style classes, it looks like you can get around this by looking at your class attribute:

class Bar(Foo):
    def foo(self):
        print "Bar.foo"
        super(self.__class__, self).foo()

Buildbot isn't using new-style classes...yet...so we can't use super(). Another workaround I'm playing around with is to use the inspect module to get at the class hierarchy:

def get_parent(obj, n=1):
    import inspect
    return inspect.getmro(obj.__class__)[n]


class Bar(Foo):
    def foo(self):
        print "Bar.foo"
        get_parent(self).foo(self)

Life at Mozilla

Well, I've survived my first two weeks at Mozilla.

It's been rough, what with the mandatory Rock Band, and the always excellent and plentiful coffee.

Thanks to John and the rest of the Release Engineering team for their very warm welcome. The whole team got together in Toronto last week, so it was great to get to meet everybody. We're a pretty geographically diverse group, Toronto actually has the highest concentration of RelEng people...with 2 people.

I'm slowly getting up to speed on how all Mozilla's systems work, but something tells me that I haven't felt the full strength of the Mozilla firehose just yet!

Moving on

I'm changing jobs.

Yup, after nearly five years at Side Effects, I'm moving on. I have somewhat mixed feelings about this...I'm sad to be leaving such a friendly and talented group of people, but I'm very excited about my next job.

I'm very happy to say that I will be joining Mozilla Corporation in their Toronto office starting in October. I'll be working in the Release Engineering group, helping to make sure that the world's thirst for new Firefox builds can be satisfied! I can't say how excited I am about this, it's pretty much a dream job: getting paid to work on a great open source project!