Better nightly builds
On November 24th we landed some changes we think are a big improvement to how we’ve been doing nightly builds.
We’ve started doing nightly builds on the same revision across platforms, and where possible, on revisions that we already know compile. In addition, all of the nightly builds will share the same buildid.
We pick the revision to build by looking at the past 24 hours of builds, and finding the latest one that built successfully on all platforms. If there is no such revision, we’ll build the latest revision on the branch and hope for the best. We also do some extra checking to make sure we don’t build a revision that’s older than the previous nightly.
Prior to this, we would trigger the nightly builds at the same time every day, but which revision and buildid each platform got was completely dependent on what time the builds actually started. There were many instances of nightly builds having different revisions between the various platforms.
These changes are a big deal because they mean that we have a much better chance of getting a working nightly build every day (or at least one that compiles!), and all the platforms will be based on the same revision, regardless of when they are run. No more guessing if today’s Windows build includes the same fixes as the Mac build!
If you’re interested in the implementation, which includes some pretty fun buildbot hacking, the details are in bug 570814. Background discussion as to how to choose the revision to use can be found in this thread.
Excellent!
I remember when I studied the problem of having different changesets depending on the platform back when I was moving L10n nightly repacks from tinderbox. Not fun!
This closes a chapter on Mozilla’s history
[...] on the same revision across platforms, and where possible, on revisions that we already know… [full post] chris chris' random ramblings mozillabuildbotpython 0 0 0 [...]
If the previous 24 hours didn’t have a revision that was successfully built across all platforms, then will the nightly be built from the same revision as the previous nightly, not built at all, or built as soon as a good revision appears?
Nice!
@Dan We’ll build the latest revision on the branch in that case, possibly with broken builds.
Has the policy been changed since bug 570814 comment #8 where it is said that if there is no build more recent than the earlier nightly which “is good” (initially, compiles) on all OSes, then no nightly will be produced until there is a build which does?
[...] Catlee landed some changes recently which mean that now: [...]
Tony,
Yes, we will always try and create nightly builds.
“If there is no such revision, we’ll build the latest revision on the branch and hope for the best.”
Well, from my experience in using this same system on SeaMonkey, it very much looks like that is actually not done and we don’t product nightlies at all when we don’t find a all-good revision.
Ah, forget my last (not-yet-public) post, seems like bug 620115 is a problem with SQLite thread-safety and your code.
[...] syncing with the Firefox configs, among others things fixing broken L10n builds and picking up same-ID always-working nightlies. Or, the latter is at least the theory, as this code conflicts slightly with our setup and makes us [...]