Last year we started using some internal mirrors at the same time as making use of the hg share extension across the board, both of these had a big impact on the load on hg and time to clone/update local working copies.

I think what we’ve done is pretty useful and resilient to various types of failure, so I hope this blog post is helpful for others trying to automate processes involving hg!

The primary tool we’re using for hg operations is called hgtool (available from our tools repo). Yes, we’re very inventive at naming things.

hgtool’s basic usage is to be given the location of a remote repository, a local directory, and usually a revision. Its job is to make sure that the local directory contains a clean working copy of the repository at the specified revision.

First of all, you don’t need to worry about doing an ‘hg clone’ if the directory doesn’t exist, or ‘hg pull’ if it does exist. This simplifies a lot of build logic!

Next, we’ve build support for mirrors into hgtool. You can pass one or more mirror repositories to the tool with ‘–mirror’, and it will attempt to pull/clone from the mirrors before trying to pull/clone from the primary repository. At Mozilla we have several internal hg mirrors that we use to reduce load on the primary public-facing hg servers.

To improve the case when you need to do a full clone, we’ve added support for importing an hg bundle to initialize the local repository rather than doing a full clone from the mirror or master repositories. You can pass one or more bundle urls with ‘–bundle’. hgtool will download and import the bundle, and then pull in new changesets from the mirrors and master repositories.

Finally, hgtool supports the ‘hg share’ extension. If you specify a base directory for shared repositories, all of the above operations will be run on a locally shared repository first, and then the working copy will be created with ‘hg share’, and updated to the correct revision.

There are all kinds of fallback behaviours specified, like if you fail to import a bundle, try to clone from a mirror; then if you fail to clone from a mirror, try to clone from the master. These fallbacks have resulted in a far more resilient build process.

A Meaningful World shows how our universe is fundamentally meaningful. Benjamin Wiker and Jonathan Witt demonstrate this by exploring various aspects of the human experience and human genius.

The chapters dealing with Shakespeare, Euclid as well as the chapters about the history of the periodic table of elements were particularly enjoyable.

One great side benefit of reading this book is a wealth of references to other great books to read!

View all my reviews

We’ve more than doubled the number of jobs to 295 distinct jobs per push, and nearly tripled the amount of compute time spent to 110 hours per push!

Here’s the complete list of stuff that happens:

Everybody knows that pushing to try can be slow. but why?

while waiting for my push to try to complete, I wondered what exactly was slow.

I started by cloning my own version of try:

$ hg clone http://hg.mozilla.org try
destination directory: try
requesting all changes
adding changesets
adding manifests
adding file changes
added 95917 changesets with 447521 changes to 89564 files (+2446 heads)
updating to branch default
53650 files updated, 0 files merged, 0 files removed, 0 files unresolved

Next I instrumented hg so I could get some profile information:

$ sudo vi /usr/local/bin/hg
python -m cProfile -o /tmp/hg.profile /usr/bin/hg $*

Then I timed out long it took me to check what would be pushed:

$ time hg out ssh://localhost//home/catlee/mozilla/try
hg out ssh://localhost//home/catlee/mozilla/try  0.57s user 0.04s system 54% cpu 1.114 total

That’s not too bad. Let’s check our profile:

import pstats
pstats.Stats("/tmp/hg.profile").strip_dirs().sort_stats('time').print_stats(10)
Fri Dec  9 00:25:02 2011    /tmp/hg.profile
 
         38744 function calls (37761 primitive calls) in 0.593 seconds
 
   Ordered by: internal time
   List reduced from 476 to 10 due to restriction <10>
 
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       13    0.462    0.036    0.462    0.036 {method 'readline' of 'file' objects}
        1    0.039    0.039    0.039    0.039 {mercurial.parsers.parse_index2}
       40    0.031    0.001    0.031    0.001 revlog.py:291(rev)
        1    0.019    0.019    0.019    0.019 revlog.py:622(headrevs)
   177/70    0.009    0.000    0.019    0.000 {__import__}
     6326    0.004    0.000    0.006    0.000 cmdutil.py:15(parsealiases)
       13    0.003    0.000    0.003    0.000 {method 'read' of 'file' objects}
       93    0.002    0.000    0.008    0.000 cmdutil.py:18(findpossible)
     7212    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}
  392/313    0.001    0.000    0.007    0.000 demandimport.py:92(_demandimport)

The top item is readline() on file objects? I wonder if that’s socket operations. I’m ssh’ing to localhost, so it’s really fast. Let’s add 100ms latency:

$ sudo tc qdisc add dev lo root handle 1:0 netem delay 100ms
$ time hg out ssh://localhost//home/catlee/mozilla/try
hg out ssh://localhost//home/catlee/mozilla/try  0.58s user 0.05s system 14% cpu 4.339 total

import pstats
pstats.Stats("/tmp/hg.profile").strip_dirs().sort_stats('time').print_stats(10)
Fri Dec  9 00:42:09 2011    /tmp/hg.profile
 
         38744 function calls (37761 primitive calls) in 2.728 seconds
 
   Ordered by: internal time
   List reduced from 476 to 10 due to restriction <10>
 
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       13    2.583    0.199    2.583    0.199 {method 'readline' of 'file' objects}
        1    0.054    0.054    0.054    0.054 {mercurial.parsers.parse_index2}
       40    0.028    0.001    0.028    0.001 revlog.py:291(rev)
        1    0.019    0.019    0.019    0.019 revlog.py:622(headrevs)
   177/70    0.010    0.000    0.019    0.000 {__import__}
       13    0.006    0.000    0.006    0.000 {method 'read' of 'file' objects}
     6326    0.002    0.000    0.004    0.000 cmdutil.py:15(parsealiases)
       93    0.002    0.000    0.006    0.000 cmdutil.py:18(findpossible)
  392/313    0.002    0.000    0.008    0.000 demandimport.py:92(_demandimport)
     7212    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}

Yep, definitely getting worse with more latency on the network connection.

Oh, and I’m using a recent version of hg:

$ hg --version
Mercurial Distributed SCM (version 2.0)

$ echo hello | ssh localhost hg -R /home/catlee/mozilla/try serve --stdio
145
capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024

This doesn’t match what hg.mozilla.org is running:

$ echo hello | ssh hg.mozilla.org hg -R /mozilla-central serve --stdio
67
capabilities: unbundle lookup changegroupsubset branchmap stream=1

So it must be using an older version. Let’s see what mercurial 1.6 does:

$ mkvirtualenv hg16
New python executable in hg16/bin/python
Installing setuptools...

(hg16)$ pip install mercurial==1.6
Downloading/unpacking mercurial==1.6
  Downloading mercurial-1.6.tar.gz (2.2Mb): 2.2Mb downloaded
...

(hg16)$ hg --version
Mercurial Distributed SCM (version 1.6)

(hg16)$ echo hello | ssh localhost /home/catlee/.virtualenvs/hg16/bin/hg -R /home/catlee/mozilla/mozilla-central serve --stdio
75
capabilities: unbundle lookup changegroupsubset branchmap pushkey stream=1

That looks pretty close to what hg.mozilla.org claims it supports, so let’s time ‘hg out’ again:

(hg16)$ time hg out ssh://localhost//home/catlee/mozilla/try
hg out ssh://localhost//home/catlee/mozilla/try  0.73s user 0.04s system 3% cpu 24.278 total

tl;dr

Finding missing changesets between two local repositories is 6x slower with hg 1.6 (4 seconds with hg 2.0 to 24 seconds hg 1.6). Add a few hundred people and machines hitting the same repository at the same time, and I imagine things can get bad pretty quickly.

Some further searching reveals that mercurial does support a faster method of finding missing changesets in “newer” versions, although I can’t figure out exactly when this change was introduced. There’s already a bug on file for upgrading mercurial on hg.mozilla.org, so hopefully that improves the situation for pushes to try.

The tools we use everyday aren’t magical; they’re subject to normal debugging and profiling techniques. If a tool you’re using is holding you back, find out why!

the long version:

RelEng have turned it up to 11 over the past few weeks to build up and deploy a new infrastructure to support the silent update program. One of the requirements for this project is that all binaries, including nightly builds and updates, be signed similar to how we already do signing for releases. Our current release signing process still requires some manual work, and it’s not feasible to manually sign each nightly build every day. We’ve developed systems for integrating signing into the build process so that nightly builds will be fully signed before they get uploaded to FTP for our nightly users to download. Incremental builds will also be signed using a different (self-signed) certificate.

Our plan is to enable signing on other platforms as soon as possible, where appropriate.

Please let us know if you have any problems with builds or updates as a result of this.

Details are mostly in bug 509158.

I’m very proud of my team on this, especially bhearsum and rail. Thanks guys!

Now we’ll be dialing it back to 10, our normal resting state.

I decided to do something about this and build something to alert me when the water level gets too low. Two strips of aluminum foil taped to either side of a piece of plastic provide my water sensor. One strip is connected to an analog input on the arduino, and the other strip is connected to +3.3V. When the sensor is submerged I get a reading of around 300 “units” from the ADC. When it’s removed from the water, a 10k pulldown resistor brings the reading down to 0.

I’ve hooked up a tri-colour LED to indicate various states, and plan to have an audible alert as well.

I’m not sure if the aluminum will end up corroding, nor if I’ll be able to power this off batteries for any length of time. Still, I’m pretty pleased with it so far!

Here you can see that LED is green when the sensor is submerged, and changes colours (like a traffic light, as per Thomas’ request) when the sensor is removed.

The primary goal of this project was to reduce the load on the main hg server where developers often experience interrupted clones or slow pushes. If things got faster as a result, that would be a bonus.

I’m pretty happy with the impact on checkout times, especially on try android builds!

It seems like most of the gains came from enabling hg share on builds we weren’t previously using them on since the update times for win32 try builds weren’t affected; the only change for them would be pulling from the dedicated mirrors instead of the main hg.

Average test times have taken a dip due to optimizations on debug builds:

Average build times have also taken a dip due to disabling PGO:

These awesome visualizations are brought do you by the Go Faster! dashboard. If you haven’t seen it before, go check it out now!

With our new rapid release schedule and the explosion of project branches, it’s become much more difficult to keep the mozconfigs in sync with code changes across merges between branches. It’s always been difficult to coordinate landing code changes with mozconfig changes; and since mozconfig changes happened out of band from regular source code changes, they were invisible on tbpl.

We’ve recently changed the build automation so that mozconfigs will be first checked for in e.g. $topsrcdir/browser/config/mozconfigs/$platform/{nightly,debug}. If this file exists it will be copied into $topsrcdir/.mozconfig, otherwise the original mozconfig will be fetched from hg.

There are two major benefits to this change:

• mozconfig changes are now a distinct point in the development history of the product itself. This means that changes to compiler versions, optimization flags, etc. are tracked along with regular code changes. Changes to the mozconfigs will trigger builds and tests like any other code changes.
• Changes to the mozconfigs will automatically be carried over between branches during a merge. If you change the compiler used in mozilla-central, then the next time we merge mozilla-central into mozilla-aurora the change to compiler will be merged as well.

Of course this means that if your branch has special settings in your mozconfig, then you need to be careful when merging back to mozilla-central or mozilla-inbound. We’ll also have to ensure certain changes like –enable-js-diagnostics are turned off when we move from mozilla-central to mozilla-aurora.

In particular, on your first merge from mozilla-central, any custom mozconfig settings you have will be lost.

I’m planning on landing the mozconfigs into mozilla-central early next week. As always, if you have any question, comments, or concerns, please let me or anybody else in RelEng know!

For all the gory details, see bug 558180.

UPDATE:
Branches currently using the generic configs (eg try, places, twigs) will lose support for mozconfig-extra[-<platform>].

I’ve been writing a bunch of simple wsgi apps lately, some of which handle file uploads.

Take this tiny application:

import webob
 
def app(environ, start_response):
    req = webob.Request(environ)
    req.body_file.read()
    return webob.Response("OK!")(environ, start_response)
 
import paste.httpserver
paste.httpserver.serve(app, port=8090)

Then throw some files at it with cURL:

[catlee] % for f in $(find -type f); do time curl -s -o /dev/null --data-binary @$f http://localhost:8090; done
curl -s -o /dev/null --data-binary @$f http://localhost:8090  0.00s user 0.00s system 0% cpu 1.013 total
curl -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 63% cpu 0.013 total
curl -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 64% cpu 0.012 total
curl -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 81% cpu 0.015 total
curl -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 0% cpu 1.014 total
curl -s -o /dev/null --data-binary @$f http://localhost:8090  0.00s user 0.00s system 0% cpu 1.009 total

Huh? Some files take a second to upload?

I discovered after much digging, and rewriting my (more complicated) app several times, that the problem is that cURL sends an extra “Expect: 100-continue” header. This is supposed to let a web server respond with “100 Continue” immediately or reject an upload based on the request headers.

The problem is that paste’s httpserver doesn’t send this by default, and so cURL will wait for a second before giving up and sending the rest of the request.

The magic to turn this off is the ‘-0′ to cURL, which forces HTTP/1.0 mode:

[catlee] % for f in $(find -type f); do time curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090; done
curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090  0.00s user 0.00s system 66% cpu 0.012 total
curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 64% cpu 0.012 total
curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090  0.00s user 0.01s system 58% cpu 0.014 total
curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 66% cpu 0.012 total
curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090  0.00s user 0.00s system 59% cpu 0.013 total
curl -0 -s -o /dev/null --data-binary @$f http://localhost:8090  0.01s user 0.00s system 65% cpu 0.012 total