Back in 2014 I blogged about several ideas about
how to make Firefox updates smaller.
Since then, we have been able to implement some of these ideas, and we also
landed a few unexpected changes!
tl;dr
It's hard to measure exactly what the impact of all these changes are over
time. As Firefox continues to evolve, new code and dependencies are added,
old code removed, while at the same time the build system and
installer/updater continue to see improvements. Nevertheless I was
interested in comparing what the impact of all these changes would be.
To attempt a comparison, I've taken the latest release of Firefox as of
March 6, 2019, which is Firefox 65.0.2. Since most of our users are on
Windows, I've downloaded the win64 installer.
Next, I tried to reverse some of the changes made below. I re-compressed
omni.ja, used bz2 compression for the MAR files, re-added the deleted
images and startup cache, and used the old version of mbsdiff to generate
the partial updates.
| Format |
Current Size |
|
"Old" Size |
|
Improvement (%) |
| Installer |
45,693,888 |
|
56,725,712 |
|
19% |
| Complete Update |
49,410,488 |
|
70,366,869 |
|
30% |
| Partial Update (from 64.0.2) |
14,935,692 |
|
28,080,719 |
|
47% |
Small updates FTW!
Ideally most of our users are getting partial updates from version to
version, and a nearly 50% reduction in partial update size is quite
significant! Smaller updates mean users can update more quickly and
reliably!
One of the largest contributors to our partial update sizes right now are
the binary diff size for compiled code. For example, the patch for xul.dll
alone is 13.8MB of the 14.9MB partial update right now. Diffing algorithms
like courgette could
help here, as could investigations into making our PGO
process more deterministic.
Here are some of the things we've done to reduce update sizes in Firefox.
This one is a bit counter-intuitive. omni.ja files are basically just zip
files, and originally were shipped as regular compressed zips. The zip
format compressed each file in the archive independently, in contrast to
something like .tar.bz2 where the entire archive is compressed at once.
Having the individual files in the archive compressed makes both types of
updates inefficient: complete updates are larger because compressing (in the MAR file)
already compressed data (in the ZIP file) doesn't yield good results, and partial updates are
larger because calculating a binary diff between two compressed blobs also
doesn't yield good results. Also, our Windows installers have been using
LZMA compression for a long time, and after switching to LZMA for update
compression, we can achieve much greater compression ratios with LZMA of
the raw data versus LZMA of zip (deflate) compressed data.
The expected impact of this change was ~10% smaller complete updates, ~40%
smaller partial updates, and ~15% smaller installers for Windows 64 en-US
builds.
Pretty straightforward idea: LZMA does a better job of compression than
bz2. We also looked at brotli and zstd for compression, but LZMA performs
the best so far for updates, and we're willing to spend quite a bit of CPU
time to compress updates for the benefit of faster downloads.
LZMA compressed updates were first shipped for Firefox 56.
The expected impact of this change was 20% reduction for Windows 64 en-US
updates.
This came out of some investigation about why partial updates were so
large. I remember digging into this in the Toronto office with Jeff
Muizelaar, and we noticed that one of the largest contributors to partial
update sizes were the startup cache files. The buildid was encoded into the
header of startup cache files, which effectively changes the entire
compressed file. It was unclear whether shipping these provided any
benefit, and so we experimented with turning them off. Telemetry didn't
show any impact to startup times, and so we stopped shipping the startup
cache as of Firefox 55.
The expected impact of this change was about 25% for a Windows 64 en-US partial update.
Adam Gashlin was working on a new binary diffing tool called bsopt, meant
to generate patch files compatible with bspatch. As part of this work, he
discovered that a few changes to the current mbsdiff implementation could
substantially reduce partial update sizes. This first landed in Firefox 61.
The expected impact of this change was around 4.5% for partial updates for
Window 64 builds.
We removed nearly 1MB of unused images from Firefox 55. This shrinks all
complete updates and full installers by about 1MB.
By using a tool called zopflipng, we were able to losslessly recompress PNG
files in-tree, and reduce the total size of these files by 2.4MB, or about 25%.
We removed a few hundred kilobytes of duplicate files from Firefox 52, and put
in place a check to prevent further duplicates from being shipped. It's hard to
measure the long term impact of this, but I'd like to think that we've kept
bloat to a minimum!
We're past the end of 2017, so how are we doing?
