chris' random ramblings

Do you want to be able to cancel your own try server builds?

Do you want to be able to re-trigger a failed nightly build before the RelEng sheriff wakes up?

Do you want to be able to get additional test runs on your build?

If you answered an enthusiastic YES to any or all of these questions, then self-serve is for you.

self-serve was created to provide an API to allow developers to interact with our build infrastructure, with the goal being that others would then create tools against it. It’s still early days for this self-serve API, so just a few caveats:

• This is very much pre-alpha and may cause your computer to explode, your keg to run dry, or may simply hang.
• It’s slower than I want. I’ve spent a bit of time optimizing and caching, but I think it can be much better. Just look at shaver’s bugzilla search to see what’s possible for speed. Part of the problem here is that it’s currently running on a VM that’s doing a few dozen other things. We’re working on getting faster hardware, but didn’t want to block this pre-alpha-rollout on that.
• You need to log in with your LDAP credentials to work with it.
• The HTML interface is teh suck. Good thing I’m not paid to be a front-end webdev! Really, the goal here wasn’t to create a fully functional web interface, but rather to provide a functional programmatic interface.
• Changing build priorities may run afoul of bug 555664…haven’t had a chance to test out exactly what happens right now if a high priority job gets merged with a lower priority one.

That being said, I’m proud to be able to finally make this public. Documentation for the REST API is available as part of the web interface itself, and the code is available as part of the buildapi repository on hg.mozilla.org

https://build.mozilla.org/buildapi/self-serve

Please be gentle!

Any questions, problems or feedback can be left here, or filed in bugzilla.

Yesterday some folks in #developers mentioned they felt their builds were getting slower over time. I wondered if the same was true for our build machines.

Here’s a chart of build times for the past year. This is just the compile + link step for nightly builds, restricted to a single class of hardware per OS.

Same machines. Slower builds. Something isn’t right here. Windows builds have gone from an average of 90 minutes last March to 150 minutes this January.

The big jump for OSX builds at the end of September is when we turned on the universal x86/x86_64 builds.

There’s a pretty clear upward trend; some of this is to be expected given new features being added, but at the same time more complexity is creeping into the Makefiles. Each little bit costs developers extra time every day doing their own builds, and it also means slower builds in the build infrastructure. Which means you’ll wait longer to get try results, our build pools will have longer wait times, dogs and cats living together, and mass hysteria!

I’m sure there are places in our build process that can be sped up. Think you can help?

Are you a build system rock star? Do you refactor Makefiles in your sleep? Great! We’re hiring!

Did you know that python’s urllib module supports connecting to web servers over HTTPS? It’s easy!

import urllib
data = urllib.urlopen("https://www.google.com").read()
print data

Did you also know that it provides absolutely zero guarantees that your “secure” data isn’t being observed by a man-in-the-middle?

Run this:

from paste import httpserver
def app(environ, start_response):
    start_response("200 OK", [])
    return "Thanks for your secrets!"
 
httpserver.serve(app, host='127.0.0.1', port='8080', ssl_pem='*')

This little web app will generate a random SSL certificate for you each time it’s run. A self-signed, completely untrustworthy certificate.

Now modify your first script to look at https://localhost:8080 instead. Or, for more fun, keep it pointing at google and mess with your IP routing to redirect google.com:443 to localhost:8080.

iptables -t nat -A OUTPUT -d google.com -p tcp --dport 443 -j DNAT --to-destination 127.0.0.1:8080

Run your script again, and see what it says.

Instead of the raw HTML of google.com, you now get “Thanks for your secrets!”. That’s right, python will happily accept without complaint or warning the random certificate generated this little python app pretending to be google.com.

Sometimes you want to know who you’re talking to, you know?

import httplib, socket, ssl, urllib2
def buildValidatingOpener(ca_certs):
    class VerifiedHTTPSConnection(httplib.HTTPSConnection):
        def connect(self):
            # overrides the version in httplib so that we do
            #    certificate verification
            sock = socket.create_connection((self.host, self.port),
                                            self.timeout)
            if self._tunnel_host:
                self.sock = sock
                self._tunnel()
 
            # wrap the socket using verification with the root
            #    certs in trusted_root_certs
            self.sock = ssl.wrap_socket(sock,
                                        self.key_file,
                                        self.cert_file,
                                        cert_reqs=ssl.CERT_REQUIRED,
                                        ca_certs=ca_certs,
                                        )
 
    # wraps https connections with ssl certificate verification
    class VerifiedHTTPSHandler(urllib2.HTTPSHandler):
        def __init__(self, connection_class=VerifiedHTTPSConnection):
            self.specialized_conn_class = connection_class
            urllib2.HTTPSHandler.__init__(self)
 
        def https_open(self, req):
            return self.do_open(self.specialized_conn_class, req)
 
    https_handler = VerifiedHTTPSHandler()
    url_opener = urllib2.build_opener(https_handler)
 
    return url_opener
 
opener = buildValidatingOpener("/usr/lib/ssl/certs/ca-certificates.crt")
req = urllib2.Request("https://www.google.com")
print opener.open(req).read()

Using the this new validating url opener, we can make sure we’re talking to someone with a validly signed certificate. With our IP redirection in place, or pointing at localhost:8080 explicitly we get a certificate invalid error. We still don’t know for sure that it’s google (could be some other site with a valid ssl certificate), but maybe we’ll tackle that in a future post!

After the graduation we had planned on spending the rest of our Christmas holidays in Courchevel to enjoy some skiing in the Alps!

For some reason instead of driving from Fontainebleau to Courchevel (a relatively easy, but long 6 hour drive), we had booked flights to Geneva via Zurich. Pick up the rental vans in Geneva, a quick 2 hour drive to Courchevel, and we’re there! Easy, right? What could possibly go wrong?

The same cab driver who picked us up from Charles de Gaul the week before met us at 6 in the morning outside our hotel in the same van he had driven before…except now we had two more people (Nat & Jeremy) plus luggage. Seven adults plus driver and two kids with luggage in an 8 seater van is…a tight squeeze! The kids ended up on our laps, and everybody had luggage piled on top of them. It was a relief to get to the airport and be able to move again!

Remember that vicious snowstorm that crippled the Frankfurt airport? Yeah, most of Europe was still getting snow and airports were struggling to cope and our flight from Paris to Zurich was a bit late taking off as a result. When we arrived in Zurich we were informated that our flight to Geneva had been cancelled, and all the of the other flights that day were full. The very helpful Swiss Air agent offered to put us on a train to Geneva instead. No problem, we said, let’s just get our luggage first. It’s about 1pm at this point.

And so began the Great Waiting.

We were instructed to head on down to the luggage pickup area and await our luggage at a special carousel reserved just for suckers redirected luggage.

An hour or so later with none of our luggage in sight (but plenty of other folks’ luggage stacked up along the walls…not a good sign!), further inquiries to the luggage folks lead us to believe that maybe our luggage got sent to Geneva without us. No wait, it’s still here. Oh, now we don’t know where it is at all…it must be lost along with the tens of thousands of other pieces we haven’t dealt with in the back.

The good side to all this is that the two boys were having a blast. No, really.

Something I never realized before was that the luggage claim area is mostly deserted. The time between when flights arrive offers two young boys a giant playground all to themselves: all kinds of interesting things to climb on and lots of room to run around and throw toys.

At around 5pm we finally give up and decide to take the train to Geneva. I think the boys were kind of sad when we finally left! However, their mood quickly improved with the discovery of a playground on the train!.

Three hours later, we’re in Geneva, at the airport, and our luggage is waiting for us! We wonder how long it’s been there…probably all day

Picking up the rental vans was relatively painless, and the drive to Courchevel uneventful. It was pouring rain for much of the drive though, which didn’t bode well for skiing. And at this point the frantic pace of the past few days really caught up with us, several of us were sick now with colds or fever.

When we run a try build, we wipe out the build directory between each job; we want to make sure that every user’s build has a fresh environment to build in.

Unfortunately this means that we also wipe out the clone of the try repo, and so we have to re-clone try every time.

On Linux and OSX we were spending an average of 30 minutes to re-clone try, and on Windows 40 minutes. The majority of that is simply ‘hg clone’ time, but a good portion is due to locks: we need to limit how many simultaneous build slaves are cloning from try at once, otherwise the hg server blows up.

Way back in September, Steve Fink suggested using hg’s share extension to make cloning faster.

Then in November, Ben Hearsum landed some changes that paved the way to actually turning this on.

Today we’ve enabled the share extension for Linux (both 32 and 64-bit) and OSX 10.6 builds on try. Windows and OSX 10.5 are coming too, we need to upgrade hg on the build machines first.

Average times for the ‘clone’ step are down to less than 5 minutes now.

This means you get your builds 25 minutes faster! It also means we’re not hammering the try repo so badly, and so hopefully won’t have to reset it for a long long time.

We’re planning on rolling this out across the board, so nightly builds get faster, release builds get faster, clobber builds get faster, etc…

Enjoy!

We had a few days between arriving at Natalia’s graduation, so we spent some time touring around Fontainebleau and Paris. The train into Paris is pretty short, about 40 minutes. Between the commuter trains and the Paris metro, Thomas was in seventh heaven!

Thomas has been excited to see the Eiffel tower for a long time now, and the real thing didn’t disappoint! It was raining a little bit our first day in Paris (the 19th), but that didn’t deter plenty of folks from coming out to see the tower as well!

There was a bit of a line up to the elevator to get to the top of the tower, so it was pretty dark by the time we got up. Thomas loved running around, looking at all the city lights, and especially being blown around by the high winds buffeting the tower that night! Martin enjoyed it too, most especially undergoing an emergency diaper change at the top!

The next day (the 20th) we started out the day by taking a short walk from the hotel to the Château de Fontainebleau

One neat piece on display is a plate painted with Niagara Falls around the 1830′s. The Falls look much shorter in the painting than they do now. I wonder if that’s what they really looked like back then?

The boys weren’t very interested in the interior of the Château, but did like running around the grounds!

Later that afternoon we boarded the train headed for Paris again. We discovered shortly after the train left the station that we’d left one of our backpacks at the train station. Desperate, we called Natalia, who was still in Fonty, to see if she could get in touch with the station staff to see if they could hold the back for us. Natalia called back to say there was no way to get in touch with the station staff directly, and that we’d have to check lost & found (wherever that was!) the next day. Little did we know…

… Natalia had already found our backpack at the station, and had taken it for some joyriding! We were reunited with our poor backpack at Notre Dame.

We finished the evening with a great dinner at Le Relais de l’Entrecôte.

The following day we spent relaxing at the Jardin du Luxembourg,

followed by a quick visit to the Eiffel Tower in the fog,

then finished the evening at a restaurant called L’enfance de Lard in Paris, which in Melissa’s opinion was the culinary highlight of the trip! She ordered a delicious steak tartare. Thomas devoured his grandfather’s ceviche-style fish appetizer because we told him it was sashimi The food here really was delicious! The owners let Thomas grab as many candies as he wanted from their giant candy dish afterwards.

I think all French desserts have to be lit on fire at some point.