We’re using buildbot to manage job queues and handing jobs off to slaves. Because of load issues, we’re currently running two buildbot masters, and each one handles a subset of all the branches. We’ve been working with the upstream buildbot project to be able to coordinate jobs between masters with a backend database.

The slave prioritization is implemented as a buildbot hook here:

http://hg.mozilla.org/build/buildbotcustom/file/eee27559b198/misc.py#l158

and used in places like this:

http://hg.mozilla.org/build/buildbotcustom/file/eee27559b198/misc.py#l618

This is way cool! Is there a good place to read up on the details of how you handle the queue of jobs and handing them off to the slaves?

Bob