Skip to main content

Extrakto!

One of my favourite tmux plugins is extrakto.

It's this amazing little plugin that lets you insert text by fuzzy finding within the current tmux session.

Why is this useful?

Think of how often you copy/paste some line of text from the terminal in order to pass it into another command. For example, the output of git status, or perhaps your build or test failed and you want to open that file in your editor (nvim of course!).

Instead of taking your fingers off your keyboard, extrakto lets you quickly find the text you want and add it at the current cursor position.

Using it is simple: just type your prefix (Ctrl-b for me), then tab, type your fuzzy search, and hit tab again to insert the text.

Pull Diagnostic Support for Neovim

I've used vim as my primary text / code editing tool for...literally decades. A few years ago, I tried out neovim, hoping to take advantage of more advanced developer oriented features like treesitter and native LSP support.

At Shopify, I do a lot of work in Ruby. Our incredible developer experience team maintain ruby-lsp, a language server for the Ruby language.

Of course, I wanted to make sure that using it in neovim was a great experience!

LSPs can do many things: auto completion, enabling jumping to function or type definitions, showing function documentation, code formatting, and displaying code linting results.

In this particular instance, I wanted to take advantage of ruby-lsp's "diagnostics" support, which provides real-time feedback about syntax or linting errors (via rubocop)

For example:

brief aside: I find the nomenclature here is a bit awkward. LSP is an abbreviation for "Language Server Protocol". neovim implements a client which implements the LSP, which connects to one or more servers with also implement the protocol. Commonly a server which implements the LSP protocol is referred to as an "LSP server" or "LSP." I'll do that here.

Push vs pull

Unfortunately for me, neovim before 0.10 only supports "push" diagnostics, and ruby-lsp only supports "pull" diagnostics.

"Push" diagnostics are where the LSP pushes diagnostic information about your source code to the client (neovim in this case). The challenge with push diagnostics is that it's difficult for the LSP to know when to generate and send the diagnostics. It has no way to coordinate with other LSPs, and doesn't really how the user is interacting with the editor, aside from receiving events when the sourcecode has been changed. This becomes a problem when many changes to a file are made; how does the LSP know when to generate and push the diagnostics? Should it do it on every change? Wait for a bit after the last change is made?

These challenges are why the language server protocol was updated to add support for "pull" diagnostics. In this model the client (neovim) requests diagnostics from the LSP server, which is much more efficient since the client (neovim) can make much better decisions about when to perform sourcecode diagnostics.

more technically, "push" diagnostics are implemented via the textDocument/publishDiagnostics method. "pull" diagnostics are implemented via the textDocument/diagnostic method

Pull support in neovim (open-source FTW!)

Last year I spent some time adding support for pull diagnostics to neovim 0.10. I learned a lot about neovim, LSP, and lua in the process!

To me, this really highlights the benefits of open-source: there was something in my toolbox that I wasn't happy with, and I am empowered (and encouraged!) to contribute and make it better for everyone! Of course, I relied heavily on more experienced contributors to the project, and, of course, my initial implementation required some follow-ups.

My first neovim plugin

However, neovim 0.10 hasn't been released yet, and so I also wanted to make sure that I could use ruby-lsp with stable (0.9.x) versions of neovim.

So I created my very first neovim plugin: https://github.com/catlee/pull_diags.nvim. This enables "pull" diagnostics for neovim prior to 0.10. It's not quite as efficient as the support built into 0.10, but it works for my day to day development. If it doesn't work for you, please let me know!

Installing it is quite simple; for example, with Lazy.nvim:

{
  "catlee/pull_diags.nvim",
  event = "LspAttach",
  opts = {},
}

Final thoughts

As a professional software engineer, I think it's important to invest in the tools that I use to get my work done. Concretely, this means that I need (and want!) to spend time learning what new tools and techniques are out there now, and to make sure to sharpen my tools.

In this case, I wanted to take advantage of a new technology (LSP & ruby-lsp) in my editor of choice (neovim). Since these tools are all open-source, I'm empowered to change them to fit my needs.

In other words, as professionals, we shouldn't accept drawbacks or limitations in our tools: we should fix them! It's just another piece of software after all!

Layoffs & Survivor's Guilt

Shopify just laid off approximately 20% of its staff. A lot of fantastic people were let go.

(If you're hiring, let me know, I can put you in touch with some amazing engineers, managers, and designers!)

For those of us left behind, there's a bit of a sense of "what now?", "why wasn't I laid off too?", or "why did X get laid off instead of me (or Y)?""

One of Shopify's values is: "Thrive on Change". We famously cancelled a ton of meetings at the beginning this year.

I like to think of these events as shaking up my snow globe.

It's natural to develop a model of how our world works, and then to stop thinking about most things except the tasks at hand. We can't be re-evaluating everything from first principles all the time; models help us simplify so that we can be effective in day-to-day activities.

But sometimes our snow globe gets shaken up. Something happens that makes us re-examine some of out assumptions about our environment. Chaos Monkeys and layoffs are some events that shake up our snow globe.

If you've "survived" a layoff: you may be feeling guilty about still having a job. Survivor's guilt is real!

If you've been laid off: know that this says nothing about you as a person. You are still amazing, talented, and can make an incredible impact on the world.

I've been on both sides of this table throughout my career. There are almost never any simple answers to the "why?" questions.

All I know is that the best path forward isn't to wait first for the snow to settle. The snow never settles.

The best path forward is to enjoy a good snowfall.

Rust learning resources

For the past 5 years to so, I've been telling myself that I want to learn rust.

And for the past 4 years, I've finished the year doing little to no rust learning :(

This year I've actually made some progress! I wanted to share some of the things that finally helped me get past the learning hump I was struggling with.

Rustlings

Rustlings is a great little project that you run locally. It presents small example programs where you need to fix up some error, or implement some small piece of functionality.

I think that Rustlings helped me more than any other tool to get over the initial learning curve, and get comfortable with the basics of rust syntax.

Exercism

While rustlings gave me a decent foundation for the basics of rust syntax, exercism really has helped build out my knowledge of the standard library. It's a great resource for learning idiomatic ways of solving problems in different programming languages. I really enjoyed trying to solve a problem on my own first, and then looking at other people's solutions after the fact. You almost always learn something by looking at how somebody else has solved the same problem you have.

Exercism has helped me build out my rust "vocabulary" more than any other learning tool so far.

Feel free to check out my profile there!

Advent of Code

Advent of code is an annual set of programming puzzles / challenges. I really look forward to doing these every year, and last year I finally finished completing all the puzzles from all the years. Last year I completed the problems with Ruby, but this year I'm going to try to solve them all with Rust.

I've published most of my solutions for previous years in my adventofcode github repo

The Book

No discussion of rust learning resources would be complete without mentioning The Book, aka The Rust Programming Language book.

I have to admit that I didn't find this terribly useful as an initial resource. Several times I tried to learn rust by starting at the beginning of The Book, and working my way through the various chapters. I never made it very far.

I did find a great channel on YouTube, Let's Get Rusty, which goes over parts of the book in order. Watching Bogdan go through various examples from the book was very helpful.

Learning about learning

What have I learned from this?

I learn best when I have a goal to achieve, and have to make use of my knowledge to achieve the goal. It's hard to learn just by reading about a topic. I think that part of that is because actually trying to write code requires that you've actually internalized some knowledge. It's very humbling to read through some documentation, and then try and put it into practice right away, and struggle to write down the most basic examples of what you've just read :)

What about you? What are some ways you've found to be helpful in learning a new language?

Managing dotfiles with dotbot

We have an amazing development tool at Shopify called spin. Spin lets you create brand new cloud-based development environments in just a few seconds, right from your CLI. These environments are meant to be somewhat short-lived; I often spin one up just for the day to work on a PR, and then destroy it after my PR is merged.

As a longtime vim and linux user, I want to make sure that all my configuration files (aka "dotfiles") are made available automatically on any new environments that I'm setting up.

There are many different ways of managing your dotfiles. I started my journey with a "simple" shell script that tried to set up .vimrc, .zshrc, etc. Inevitably this script became overly complex, and was too cumbersome to maintain.

I briefly toyed around with ansible for managing my local system, but I found it to be overkill just managing dotfiles on remote systems.

Finally I discovered dotbot, which has a few features that I like:

  • No dependencies
  • Add into your dotfiles repo via a git submodule
  • Supports most of the stuff I need, without being too complicated
  • Configuration is easy to read & write
  • Safe to re-run

In particular, to set up neovim, I have something like this in my dotbot configuration:

# install.conf.yaml
---
- defaults:
    link:
      relink: true
      force: true

- link:
    # This line symlinks the astronvim repo from the dotfiles checkout 
    # into ~/.config/nvim
    ~/.config/nvim: vim/astronvim
    # Link in our user configuration for astronvim
    ~/.config/nvim/lua/user/init.lua:
      path: vim/user_init.lua

- shell:
    # Install all neovim plugins with packer
    - nvim --headless -c 'autocmd User PackerComplete quitall' -c PackerSync

Since setting up my dotfiles with dotbot, I've discovered that I'm much more willing to experiment with different configurations and tools. It's easy to blow away local directories, or git restore in my dotfiles repo, and then re-run the install script.

One small tweak I made to the generic install script is to split up configurations by operating system and environment. I have different configurations for Linux, macOS, and spin. I also have the install script send logs to the home directory, which makes it easier to debug when something has gone wrong. Adding exec &> >(tee -a $HOME/.dotfile-install.log) at the top of the install script copies the script output to a log file in my home directory, as well as outputting to stdout/stderr.

My dotfiles can be found on github.

Learning Ruby as an experienced Python developer

As I mentioned in my previous post, in my new role at Shopify, I've been doing backend development in Ruby. Previously I had been working nearly exclusively with Python for over ten years, so I was a bit nervous about the move.

In addition to my regular work, I also tried to solve other types of problems using Ruby. Advent of code was a really great way to learn a new language.

After nearly two years in the new role, I'd like to share some of my experiences, hopefully as a way to help and encourage others who would like to learn a new language, or are worried about moving into a new role, but don't know some specific technology.

Early pains

The first few weeks with Ruby were pretty tough. Luckily, Ruby shares some similarities with Python that make it a bit more approachable at first glance:

  • Dynamically typed, interpreted language
  • Class / method syntax is similar

Method calls don't need ()

One of the first things that tripped me up was not understanding that using () to call a method is not required in Ruby.

def greet
  puts "Hello, world!"
end

greet
# => "Hello, world!"

In fact, () are optional when passing arguments as well!

def greet(name)
  puts "Hello, #{name}"
end

greet "Chris"
# => "Hello, Chris"

This can be used to build some very nice DSL features, resulting in code that is much more readable. e.g. Rail's delegate method

require "rails" # to get delegate
class Message
  def greet
    puts "Hello!"
  end
end

class Greeter
  delegate :greet, to: :message
  attr_accessor :message
  def initialize
    @message = Message.new
  end
end

Greeter.new.greet
# => "Hello!"

Implicit return

Like Rust, the result of the last expression in Ruby is used as the return value of a function.

def add_one(x)
  x + 1
end

add_one(2)
# => 3

Strings & Symbols

Ruby has a concept called symbols, which are a kind of identifier using the : prefix. They're often used to reference method names, since you can't get a reference to a method just by accessing it by name like in Python. E.g. obj.foo will call the foo method on obj in Ruby, whereas it will give you a reference to the foo method in Python. The equivalent in Ruby would be obj.method(:foo)

Symbols are also used for named parameters in method calls. E.g.

def greet(message:)
  puts "Hello #{message}"
end

greet(message: "world!")
# => "Hello world!"

However, where symbols presented me with the steepest learning curve is how they're handled in Hash (i.e. dict) literals.

a = "key"
h = {
  a: 1,
  a => 2,
}
# => { :a=>1, "key"=>2 }

It's extremely easy to get the two ways of defining a value mixed up. I've wasted an embarrassing number of hours on bugs caused by mixing up strings and symbols as the keys in hashes.

Range expressions

Ruby has nice built-in syntax for ranges:

(0..5).to_a
# => [0, 1, 2, 3, 4, 5]
(0...5).to_a
# => [0, 1, 2, 3, 4]

These are super convenient, but I almost always forget which form is inclusive and which is exclusive.

A-ha! moments

Blocks

Before really learning Ruby, I remember trying to read up on what blocks were...and not really getting it.

The best way I can come up with to explain them now is that they're a kind of anonymous function / closure.

Part of my confusion was not understanding that there are a few ways of defining and calling blocks.

These are equivalent:

mymap(0...5) do |x|
  x ** 2
end

# is the same as

mymap(0...5) { |x| x ** 2 }

The block's arguments are passed via the identifiers between the pipe (|) symbols.

In both cases, the mymap method is being passed a block, which in our case gets executed once per element (but that's completely up to the implementation of mymap). The block can be named as an explicit function parameter, and this could be written as:

def mymap(obj, &block)
  result = []
  for i in obj
    result.push(block.call(i))
  end
  result
end

The block can also be passed implicitly (check using the block_given? method) and called via yield:

def mymap(obj)
  result = []
  for i in obj
    result.push(yield i)
  end
  result
end

Once I wrapped my head around blocks, I found them to be very useful!

"&:" idiom

Ruby has this really neat shorthand for creating a block that calls a method on an object: "&:". It's used like this:

["hello", "world"].map(&:upcase)
# => ["HELLO", "WORLD"]

There are a few things going on here:

  • :upcase is a Symbol referring to the upcase method on the object
  • & tries to convert its argument to a kind of closure using the argument's .to_proc method. Symbol#to_proc returns a closure that calls the given method on the passed in object.

The net result is something equivalent to

["hello", "world"].map { |s| s.send(:upcase) }
# OR
["hello", "world"].map { |s| s.upcase }

Brian Storti explains this in much more detail in his blog post.

Enumeration

Ruby has fantastic enumeration primitives built in, just checkout the Enumerable module. Most of the basic container types in Ruby support Enumerable; when combined with blocks, this makes filtering and transforming data in Ruby a joy.

It also fits my brain better than Python's generator / comprehension expressions. When writing Python code to transform data, I often found that I was writing some code, then backing the cursor up to the beginning of the line.

In Ruby the data and logic flow from left to right, which makes it easy to chain thing together.

(0...5).map { |x| x ** 2 }

In Python the logic is on left but the data is on the right.

[x**2 for x in range(5)]

If I want to get only the square numbers that are even, I would simply add this in Ruby:

(0...5).map { |x| x ** 2 }.filter(&:even?)

Whereas in Python I would need to introduce more code before/after the expression I already have to achieve the same result:

[y for y in [x**2 for x in range(5)] if y % 2 == 0]

tl;dr

Ruby is a really nice language. I'm glad I've had the opportunity to learn it!

Python has a philosophy of "There should be one-- and preferably only one --obvious way to do it.". I think this helps with Python's readability at the cost of expressibility, elegance, and conciseness in some cases.

In Ruby there are often multiple ways of achieving the same thing. It has a richer vocabulary for expressing ideas in code. This richness allows for more elegant code in many cases, although this perhaps requires more effort on the part of the reader to understand the code.

GCS Performance Experiments

Recently I've been digging into some performance optimizations with some code that transfers dozens of assets into a bucket in Google Cloud Storage (GCS)

The code in question is using Google's Ruby gem, and uploads a set of files in sequence, something like this:

require "google/cloud/storage"
storage = Google::Cloud::Storage.new(...)
bucket = storage.bucket(...)

files.each do |filename|
  bucket.create_file(filename, filename)
end

For this naive implementation, it takes about 45 seconds to upload 75 files, totalling about 350k of data. That works out to about 600ms per file! This seemed awfully slow, so I wanted to understand what was going on.

(note that all the timings here were collected on my laptop running on my home wifi)

Jump down to the TL;DR for spoilers, or keep reading for way too much detail.

Simplify the dataset

First, I wanted to see if the size of the files had any impact on the speed here, so I created a test with just 50 empty text files:

$ touch assets/{1..50}.txt
$ time ruby gcs.rb !$
ruby gcs.rb assets/{1..50}.txt  0.61s user 0.20s system 2% cpu 27.951 total

That's still about 28 seconds, or about 550ms per asset, which seems super slow for a set of empty files.

Start digging

Looking at the http traffic generated by running this code, it appears as though Google's gem is creating two(!!) new TCP / https connections per asset. The default behaviour of the gem is to use a "resumeable upload", where one connection is issued to start the upload, and then a second connection to transfer the data and finalize the upload. It doesn't appear as though any connection pooling is happening either.

It looks like there's room for some improvement.

A few ideas came to mind:

  • Use a single request per asset
  • Use connection pooling
  • Run requests in parallel (using threading or async)
  • Use HTTP/2

Single request per asset

The GCS API is pretty simple, so it's straightforward to implement a basic upload function in Ruby.

def upload_object(bucket, filename)
  Net::HTTP.post(
    URI("https://storage.googleapis.com/upload/storage/v1/b/#{bucket}/o?name=#{filename}"),
    File.read(filename),
    {"Authorization" => "Bearer #{ENV["AUTH"]}"}
  )
end

This uses one connection per asset, and brings the time down to about 11 seconds for our 50 empty assets, which is less than half of the naive version.

Re-using the same connection

Net::HTTP supports re-using the same connection, we just need to restructure the code a little bit:

def upload_object(client, bucket, filename)
  uri = URI("https://storage.googleapis.com/upload/storage/v1/b/#{bucket}/o?name=#{filename}")
  req = Net::HTTP::Post.new(uri)
  req["Authorization"] = "Bearer #{ENV["AUTH"]}"
  req.body = File.read(filename)
  client.request(req)
end

Net::HTTP.start("storage.googleapis.com", 443, use_ssl: true) do |http|
  files.each do |filename|
    upload_object(http, bucket, filename)
  end
end

This runs a bit faster now, in 8 seconds total.

Parallelization

Ruby has a few concurrency models we can play with. I try to avoid threading wherever possible, and use async libraries. Luckily, Ruby's async gem handles wrapping Net::HTTP quite well:

Async do
  barrier = Async::Barrier.new
  files.each do |filename|
    barrier.async do 
      resp = upload_object(bucket, filename)
    end
  end
  barrier.wait
end

Now we can finish all 50 uploads in about 1.3s

HTTP/2

There are various options for making HTTP2 requests in Ruby. One we can use is async-http, which integrates well with the async gem used above. Another gem that's worked well for me is httpx.

Async do
  barrier = Async::Barrier.new
  internet = Async::HTTP::Internet.new

  files.each do |filename|
    barrier.async do
      resp = internet.post(
        "https://storage.googleapis.com/upload/storage/v1/b/#{bucket}/o?name=#{filename}",
        {"Authorization" => "Bearer #{ENV["AUTH"]}"},
        File.read(filename))
    end
  end
  barrier.wait
end

This finishes all 50 requests in 0.4s!

Looking back at our initial data set which took 45 seconds to run, we can now do in 0.9 seconds.

TL;DR

To summarize, here are the times for uploading 50 empty files to a bucket:

Method Time
naive (google gem) 28s
single request per asset 11s
single http connection 8s
async 1.3s
HTTP/2 0.4s

I'm really shocked at how much faster HTTP/2 here is.

Consumers of Google Cloud APIs should take a look at the libraries that they're using, and see if they can switch over to ones that support HTTP/2. I'm curious if other client libraries for the Google Cloud APIs support HTTP/2.

Can the Ruby gem support HTTP/2? There's an open issue on the github repo to switch the underlying client implementation to Faraday, which would allow one to use an HTTP/2 aware client under the hood. I've started working on a draft PR to see what would be involved in switching, but there are some major barriers at this point.

Hello Again, World!

First off, hello, it's been a while!

Since my last post over three years ago 🤦, life has been pretty busy.

First, a global pandemic.

Next, we welcomed a new baby into the family the summer of 2020, right in the middle of said pandemic.

Then in the fall of 2020, I resigned from my job of nearly 12 years at Mozilla and I took on a new role as Staff Developer with Shopify. I've been doing backend product development work, focusing on making our themes more customizable by merchants, and extensible by third party apps.

I even got to publish a video describing a new feature I helped to build!

Shopify Unite 2021

(Look, I'm on the internet!)

I've really enjoyed the move so far. It's been great to switch gears from infrastructure work in Python to product development in Ruby. I intend to blog a bit about my experience learning Ruby as a relatively experienced Python developer. tl;dr - I'm glad that I did. Learning new programming languages and tools is almost always beneficial, as it helps you to think about problems in different ways.

Updated GPG key for signing Firefox Releases

The GPG key used to sign the Firefox release manifests is expiring soon, and so we're going to be switching over to new key shortly.

The new GPG subkey's fingerprint is 097B 3130 77AE 62A0 2F84 DA4D F1A6 668F BB7D 572E, and it expires 2021-05-29.

The public key can be fetched from KEY files from Firefox 68 beta releases, or from below. This can be used to validate existing releases signed with the current key, or future releases signed with the new key.

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFWpQAQBEAC+9wVlwGLy8ILCybLesuB3KkHHK+Yt1F1PJaI30X448ttGzxCz
PQpH6BoA73uzcTReVjfCFGvM4ij6qVV2SNaTxmNBrL1uVeEUsCuGduDUQMQYRGxR
tWq5rCH48LnltKPamPiEBzrgFL3i5bYEUHO7M0lATEknG7Iaz697K/ssHREZfuuc
B4GNxXMgswZ7GTZO3VBDVEw5GwU3sUvww93TwMC29lIPCux445AxZPKr5sOVEsEn
dUB2oDMsSAoS/dZcl8F4otqfR1pXg618cU06omvq5yguWLDRV327BLmezYK0prD3
P+7qwEp8MTVmxlbkrClS5j5pR47FrJGdyupNKqLzK+7hok5kBxhsdMsdTZLd4tVR
jXf04isVO3iFFf/GKuwscOi1+ZYeB3l3sAqgFUWnjbpbHxfslTmo7BgvmjZvAH5Z
asaewF3wA06biCDJdcSkC9GmFPmN5DS5/Dkjwfj8+dZAttuSKfmQQnypUPaJ2sBu
blnJ6INpvYgsEZjV6CFG1EiDJDPu2Zxap8ep0iRMbBBZnpfZTn7SKAcurDJptxin
CRclTcdOdi1iSZ35LZW0R2FKNnGL33u1IhxU9HRLw3XuljXCOZ84RLn6M+PBc1eZ
suv1TA+Mn111yD3uDv/u/edZ/xeJccF6bYcMvUgRRZh0sgZ0ZT4b0Q6YcQARAQAB
tC9Nb3ppbGxhIFNvZnR3YXJlIFJlbGVhc2VzIDxyZWxlYXNlQG1vemlsbGEuY29t
PohGBBARAgAGBQJVrP9LAAoJEHYlQD1/DRWxU2QAoOOFRbkbIU1zKP2i3jy/6VKH
kYEgAJ9N6f9Gmjm1/vtSrvjjlxWzzQQrkIhGBBARAgAGBQJVrTrjAAoJEMNOV0fi
PdZ3BbkAoJUNHEqNv9dioaGMEIpiFtDjEm44AJ9UinMTfAYsL9yb15SdJWe/56VC
coheBBARCAAGBQJWBldjAAoJEAJasBBrF+oerNYA/13MQehk3AfkljGi252/cU6i
1VOFpCuOeT7lK2c5unGcAP0WZjIDJgaHijtrF4MKCZbUnz37Vxm0OcU8qcGkYUwH
i4heBBARCgAGBQJVrSz+AAoJEPCp59zTnkUulAYA/31nYhIpb7sVigone8OvFO19
xtkR9/vy5+iKeYCVlvZtAP9rZ85ymuNYNqX06t+ruDqG2RfdUhJ6aD5IND+KD5ve
7IkBHAQQAQIABgUCVaz9fgAKCRCzxalYUIpD8muMB/sH58bMSzzF9zTXRropldw7
Vbj9VrRD7NyoX4OlDArtvdLqgPm0JUoP2gXINeSuVPpOfC676yVnBEMjIfqEjq09
vcbwayS+Ncx4vQh2BmzDUNLE3SlnRn2bEWr9SQL/pOYUDUgmY5a0UIf/WKtBapsP
E+Zan51ezYSEfxDNfUpA4T2/9iWwJ2ZOy0yIfLdHyvumuyiekJrfrMaF4L9Q0OnJ
wp1PwkvN4IVwhZeYDtIJN4nRcJK5LrwU7B97uef2hqBBll7/qCHl5y4Khb0csFan
Ig+pQLPUJdIiYtzoFtlgykB61pxqtU9rqGKW02JzEUT8DdPUXxmMBy6A8oGeBRH/
iQEcBBABAgAGBQJVrRdcAAoJEGVzgtv/JREKQJgH/3nD/3/SumL7nG2g7Y1HQqWp
hUbn40XWvjZcHq3uBUn1QYXeZ5X56SANLM2t+uirGnNaZXW3cxEl5IyZVLbmcLWE
BlVAcp2Bf3FXFbdJK59f+M+y2+jZT9feTyrw+EtLoiGTxgkLdJyMyI0xGmQhMx5V
1ex1CxhZK2JPjzCVYriBI0wIbmKi90YNMQoSsdMhYmX9bHl6XWS9TCDWsqj25FLY
JL+WeVXpjO0NjRwEE6pc/qldeJYG5Vbf0snGxIerXe+l5D8Yd4PEAnpj58+5pXeo
GYZn3WjX8eTFMAEU+QhLKWQ+j/Y8Kijge7fUxnSNBZ2KEnuDN/4Hv/DrCFLv14CJ
ARwEEAECAAYFAlWtZVoACgkQ5DJ8bD4CmcBzsAf/RMqDdVHggQHc0/YLt1f/vY9Y
7QQ6HwnDrtcNxxErSVcMguD8K6Oxir0TMSh+/YuZAW8K4KSgEURwZqz4na8/eOxj
8bluNmlcAseQDHswqU6CyB95Woy3BocihH7L0eDXZOMzsa33vRQHBMioLxIbpnVt
VbFR1z7tmyfjcOrzP32xo5QoPoczKX26luMBjAvbw1FC0is2INnmUSYM4uH7iFZu
XGPFYxcAqODqy5ys3MoPa4oZ71d0HoiRil1+s0Y+2ByddZ19pE2TXp4ZXNYNUj/2
aRj8b4sTjR4rqhHIx/vfoK+VCNy/skFUZOyPdbbymE0stTRSJ1gr9CZLcBWYF4kB
HAQQAQIABgUCVcFZcAAKCRCJFz+VfFX5XqApB/938p+CJiDRnh2o7eDWnjSyAu7F
WmWGkOQnjI/kraKx1vojsYnKRXD6mjq1QJ8Hsp4taJnLQjcokNTUiST4m/e4ZJEx
PWuJKkwlralWGH6NpqYcgWPajSYb0eYQC4YqS0kfyzolrHdKI8Y4NGEU7yy5zsHw
WkHt/mpNQMrYnXwyWdIrc03X/OXo51dJyshJDRw3InREyBblFJcLvArNHz219wMr
XAicPytw4wfPpVrmDx6GrZcI8q8ECWCjwSXXv7hRpEuFLSy5XPhMc+wYBJjNlUoi
FBAF/7zENd3rMn9SCQLiIFYe0ubmO+bpeGy7TizbxOaCIfgUouyy0BQXNuJBiQEc
BBABAgAGBQJV0hrqAAoJEK18uZ+CSLoPzEIH/1D6sJMNAJtZCRGhJXvv6SYhv4pU
VNyDF9FnUvRsovliojoe4IkuBTWKhPGrxbiD5IO/izr38shqNhhm9JE2/SQZHObY
Pi+lyfDKbJgImTNxmS4F7JHnRLr37VxK1sVvuNkynJnqvCcp1g5xwNIx1rKcka3i
uqJj6toM8XQfgsTHH1rUkWHbUV3QwNzXm+yhFm2s6QzxBooPzmFn8AY7CXD4pvcM
R+M0Zy+e42nngd8lzRnmTBVig4pRq0GCMulFG+XjeVQZFpoIIxo2k1lczbRmGttO
NdGWSjxBUxReoTbSwM3C/50NrobycGQgY0gd6LGtWtU8/uEfklEy2NluxYWJARwE
EAEIAAYFAlWtAUYACgkQVu5xjc4OFUs0OAf+LM0dyyvUFGdXfJDpP2xMknXzsHAX
WFEtH5jein58mv6dD3fTVcCouo1vMQH3WFFSLYZvwtNnHGrSBqFbNKqZ0ATQ5tcY
aWsSZ+MVJJMXJDXFG/Oihg1nNOM33VdfV0RGPKP1I4cEROxms3TUFkHW3cSCgMzs
8I1OxfSoLrm6da8EN+2ct2InqzdQL2yisyTyrdmXoNpwXDxApKYkvVHQ4+9eJI5m
0ZAr0mBjIeJdATcw4/lIVKTrV7UhrChxiffYJcz4SSC1crmr+2Fzw53CyAsAmYal
UHep3Yr05oQ4oJRX9X3VrY/yELHwwxXaxCAdwwHbbXAMhZsPk9Mc20J6BokBHAQQ
AQgABgUCVa0isQAKCRCj1lIXO3Y+j6ZeB/91Q9/qr5oMWgOMsix8kflBLw2f/t+t
RR0SWDw90bG1npJB6nq5Hl+Bz4/A4SWFTFrrrlZi1Enjn1FYBiZuHaSQ/+loYF/2
dbQDbBKShfIk3J0lxqfKPAfKopRsEuxckC8YW1thGxt5eQQ8zkJoqBFTBzwiXOj3
/ncJkX9q9krgUlfTSVmrT9nx0hjyNQQXrghsmBtpR7WCS7G7vNRGCNUorhtviUvL
+ze1F7TTSGspVsVxo2ghmz5WT/cD9MV1gcVjojYmksh5JIl39jCHr9hl8aRId/Of
zsN+TKuBcpAxDkm9BCAps7oY8FlLKDFZTtHa000AkodKHT88nwnvKuqPiQEcBBAB
CAAGBQJVrTkDAAoJEPbQ92HczOykK9YH/0MARo3HlYXeS2bDqM/lwK/rQcPCCyYk
e6wbICjncbCOjgXHqG/lBhClNs7hp/7gqkUaR7H5tmeI4lalP40mSHHnnFvMD3Tc
yhn350igK0bgrjWQDaYxhKlHT3vIXd/C24/vRSAxmqIKbP+IoXOyt2GMTQq8GOm2
dgYRaTkwyHnGWnMaibctX8D4oCYR0/D4YJqPkfqobf8+1ZfP5GaMbSxE/Jwdo0kJ
a4vPjEzFXbygAbncapzdwN6zgel2zh885rz7B7vIpMr/Y7eV85Q68qdyyhLe8cL8
Y18YPzpFf+/PZNbgYxouafvnFwBhPQwg0gUF/+1eM3UE2ua+saSTGduJARwEEAEK
AAYFAlWtCVsACgkQM0LhtmejiGMovwf8CfYJHNbwiwSMUoP4n7FrmElhBtxvlbnC
MZKz08v+lFsfS3wU1LUN69GqirfF0vkQRSlSBp7niCLHQCfSoqHMLgxF0P2xgXLj
aYM/t/rxXDawJmW18G04dqFrtCPZTbwMT2PsPHTiWQdaN0e50lXk9Vo+l6VbwQMg
4zH7icZadeJgQooxFalHYFVXUVeex9t8/YdanFVrHFa3tao6azBTSUkJvZtIu14S
fxigDWIIwsx0xpVfJf3a/xC6HY3Q1a3NeBz3i6DwaK5wYqijZKl0WVdULKyqU98o
F6y0mUv3d2o/p07Cqgeo6xxMkHqu83OLa2a0C7tYPLgL4EFc2FtikYkCHAQQAQIA
BgUCVaz7KAAKCRCWO3gxCjexfKxrD/4npm1rB7+pPlotbqK37Mur7egPbVSAzVNU
/zUKPAuGUeP3C64YN77ETx1kDuS+meAqMDHFc9Bf8HivPbtj6QcK96U5KstbmSh1
Ow9YiQtxJgxGjg/CzREgZAFcjy0MhoklyPsFhv07s6MLOJMSM/krEN5nqjifQ0Wd
mTk02FLoHVWcLdjfgMiPiSjGbU3k7luvjPyRNzk831szE5mfa74rEYh4TBklse+2
uB4DFQ/3oHZ1Sj6OBK6ujmNKQjIP7Cl+jmjr7+QK0OJcRaj/8AckDA5qXTZACh1S
2syCDDMnX0V+dTxGCIoWOK+tt9mLohMzpEeD4NIX4qdpbbCRzeYZMHSomyBIsbA6
B+/ftDE7W1N0/FtJ9adkkCynKULvh2CH5c5hgOOL22M+2spnywRoeJRUWU7hBM5O
UH3JjA4Tu4j/cwp7dD7QzZrzmC9f5LQJ3OelejvVowWPQd3/tky4o1q6wlmFqAcA
gtu97UwgBOSR9sJPGDlt1iC91UYAiBQQAA7ya8uXUS84mCQwTlr8j+YrowvEHK4I
xpPREytT1LzzV/4Am4ndDFtujy83QjL0qaIIim1xIwoEosd4yidhpczw7f3b9dQp
uBIFeQuhM7JsxP4tmE7S6k6GlEmqa3INPVaPGnsUGS7+xSMlcJXLtimPCSQvFma9
YiGV5vtLy4kCHAQQAQIABgUCVaz8uAAKCRASy06X4H5n0dg0D/9QoxIh9LRt1jor
7OHG4xKUjKiXxn/KeQNlJnxI55dlWIvJEJGheFjaDomzKBYuxmm2Ejx+eV5CHDLU
YsLFYwWf8+JGOP75Ueglgr8A0/bdsL63KX6NP2DCg8XR4Z1aeei3WMY7p/qMWpqb
QoAv9c3p49Ss2jSNuthWsRR6vbQ9iwze2oaUaA44WKQyhhbCwBU4SHYjlKCLqIBh
/HXZFhZ4rDfuWgPBKvYU1nnOPF0jJRCco3Vgx3T9F+LZ3zo5UPt1Xapr3hMVS9ia
Jyl1w4z2miApUaZuHPuWKuO4CJ1GF1mS5T6vG8gB3Ts5zdtBF2xQIkCz+SM7vW/2
i/82oq6P8EuLHEhrQPR4oTjXIvXdEJ9kgbjqcj8Xk+8teEOnuwh6iEhay9i/bf0D
3Jd+roFN5dnWPxhOVjzrI3fwlK1/ylsZYqUYBEzt7Wj0MdhjeKssI5YICcqYXXjB
ttMw4B7DZXPFXzz3kHB56jZ/II4YUjpLO85Jo5A9SV+aIqa0mvCt6DvVWy/rhfxf
oUdqNlhX11gkVLaA7xxgn/NqPOf+h5hVO2mwWkmart9YHKMZ3ukCdke65ITL/nsY
Sm2ZhG7OYjaCfu9jPWtkBstOEWyT9q4JTdViR7wN3eMefEG6rb49rxOYvGJu+cTV
kp3SCpl0w1j+tPj4tkj7ENzPMXdnuYkCHAQQAQIABgUCVa0s4gAKCRCKsTKWOgZT
euMyEACKOySKAd/xDcPcHg7Prvdws04Z8DIR0dY2qUlbRVx2jTmIXyry63CqbOJF
bDg9uk5x0+lSotvrWtZ+NKSrg9VM6vyV4cc2P9rhqIBi3wO2elzAmpOaS2KKOjQ+
2fS/xqh91ElJUu09xXQXJ0vMrqgui+zN1YBDiJV0WOmm90Mm2NPiihcWZmBmDorO
qMQabwbjBLi0yUVHgAlkilY3mAB4tmEKDeN+4pYSAAhXAll9U+nyoVMgwMJscZya
zOp4MqMbmFjyr4p5AGzv+OOJtjtCNKT6oW9Y+URLY0YKeOsPk0v5PlbQCVBlLeSB
sNZudKav/Gvo7Mvz5uLTcneBFb+haYIiXO/FQm4uBHkzdNFLgaph81Wzh62AhbtB
lfBOj/lbzN3k/xRwo64QU+2Z9GOhFlhjfROquY70FCQcspwNuqCdZybnkdpF2Qrr
6Pi0qKR/Xb9Vd7PW0/gKQdwwlYTiDemgA21mYeJrYw873/7U/+kLFRvmPAEX4IOI
OEN6XVjxvu78REi6CmXxOoYnH4aRSXDRyi1nsGjB43AtfAMMNCUigDgFP4sUsZAG
1RAoxBhOsO/g9S5wx8H3rKITCXDjQh2SYeBwHFcU03EMcyzEQhbZNighN+aRKGIi
bteRxISiKU+kcWaHolemeo6wGF87QXEpJaQ2OwIoIxQYvDDmQokCHAQQAQgABgUC
Vaz/8QAKCRA/8xuvEEv54t06D/9n1Nyn2QSUN1mXd7pomoaka+I2ogDbQpu9iuFq
bkqfcH3UuG8yTKlPp9lYDBs0IEfG85Js6iVxJIultocrcDmOyDkyEsnYbdel/tn3
X4yqD8eI6ImRoCE+gnQ3LoEIHuODfJoosM/jAHANs4fsla4/u5CZDXaaq7pYXGiT
t7ndsfmLiCa7dAg7bVFfJagsnL/VjlfeWM9nW01rDL9LPxSN4tq7ZKXWZDonFZYJ
4unsK/Cn6Pqco4Wb+FUOWCcWt8in1pgeNHZ9WnAgXG999/3iCbbQTLB6uVwY4Ax5
P7VApnLVXV6QFVf7bN1DxE8kZk+pfLGcuD1LJSF0skE80M17kAt+iV+fam8EYzeG
dG6cY6w+srndaMaq9ddiHIiQkR35SjJAGnrNRj8ooUr/vKOBnFfuwJLA2MOUVPZ8
HWB+WXW8qhihw9CXa38Hdt4o5knMGRIyTWEF0TQDtRGQ6hisVBN3OxJRXBj7/QgC
G/GoYpweGKcsMU43p57TzbnXVVUytJsLFyexOGNzrUIxgDVPEvTUnNvdAihNZPdb
W3YdFkP9pdwOyDpQwebXELUx1kp4ql0laueex4L1v+0a6rDYQeK1gOq5UGY+THRS
gB2xsHl5zeryfgnjlUkUlxKuumz+9FI2fRtSpxmWllJkRF2oFMGRuLPGAWe8nHvf
gkuGVokCHAQQAQgABgUCVa0bowAKCRCVY0f2+/OkFWKREACZ9TOmzvY6mrfWVEdl
dcYPj8cU/1LJhGdbNo5YYMx+A72nchxGXepHA65OEK+f6rFMeZFPwpQPy6Sj3MhT
623H/PECfeG87WcLOyJbfc3i9T5jvxS+ztG6abYI2J/50oMvjUWdWkDX3VvdPc0Z
Z+KC+oHvx9a/9Yki48m4CEKglgVsrRW/b9AXZQCj07bB0GjQQtkqY/m1Z8m4ttzx
fO7OBo/jHNF2An4/4gUDirXNDj0UdB5FYFJaTEUCneIj2x0fk1r4u6na8tINhiZ0
M7IgjnDlBD5jwzvwG+3kYE6TnYp9Mfeg2MPC13tp7jrJatLLutrOzvmSVLGLXbkh
9w+v+vx7qO3TxZUNlFqTmYs+vI2V/9j7KYV7Ttoind6Io7X9ImnYrvd8JOyVcO38
67MplKnrnqHJvFStE+JcHEcw5aRw+WVmoFd/obGc34V3K62T977QQGOkrTYDEdje
KADfjXXZkZMZc0IvzLBOJ1XB45+PKqJYCcJJS8Xr55+NGCDaaUPWDpkNGIqmX2n9
kYROMKG6uWkZIqG0JlZkga3THSJIvLiy6uoOvDC4GoQ9JnTwpGv6r1Hwcg+4DCOr
YKOoPKMMU24vHx2FtRRUgCXtr2cmi2ymHlUrtz8EXS4tblic8lixcbvPUqLEvbJ2
gfWQvjXNd1whYE/wfvI9WBTEIokCHAQQAQgABgUCVa0b3wAKCRC8FzAbSRs/IQhX
EADiKbCnsN/+Plllxn6SQHACEU75ackx+Q02XiD/u+wUptYUGmJi4aaW9f6mgzed
OxYK4S+/dCiFtkcYlL+FjaR0C7G6tMjrDgW+8nQCTPUNQA0gX2B8n06a7Zmdv3Eb
V/PIJJwTNSBp/dqKbvPKnRquOOpH+ayZ3awKOq/LlWBErbW1gB+FabN0lCe0iUIQ
TF9OH3GC4QsMtIrePueBmVrVPcHATV2Vw9UPqX1uX/tlXm5eai06oVT7V0FwUbg0
o1eacblNXvHciHpe33zZIKkGBWwSjDVcU9/SN+U8GfoMYmyCma4iN3KaCklpzBkJ
iQZtNKPAB5KJti8LDUxFi2sJd3sqWaZDGFhO+/PKhBKpqIhAzx1ppd11zLgh0eg6
gQlXN8D8ELISRvQqGGNNZdChEFdzGElg5SMfmeEd37OaX4wceLLV0v7EA0doHMVo
0enFhSwU3YwtwxbiukKc7H/ylG7+jvntjY+z7KktRsY/FkklrbrNhddMBQMMSAQU
Uz1GJ+6NUKmzXjqxFuuh3OAhqNzhJyABZWQcNMph+rogEslkenwoHV9gWRWtS3CM
ybJkKkbsWpYhMZNY6hFtgCwida7NPs8369v+yTTE6TU/NIlXUKYIf2LMqtOpEBTj
aN3jKpUi5DeE3zBeh6iVKUrfCXbt8O0rYQPNWGSW+MZ2t4kCHAQQAQgABgUCVvA4
GwAKCRBE9G4UbQI5XfS9D/9XPK7jg0lmsNZ2sDIyeAw5n6ohSR5F20ocTMAVeXqN
7VkvJdNpIqHJa13EP408DgTy9BsSptym/OQGE6B82BU7FZTEL6eMHnGGDg+5ktx9
+b73xLedzK75ti6ED+QuA4kDYcvW8hASht0zRcmFUzwbtuEopJ1Lk1R3oFLwCAov
lhduC45nANWrTK5U+D1U2obl5PAvx+9mEfgvojlGH/C/WD74W+cQZFH7t4+muRza
mckLyPftnTxjNF/lpYIm7z0QOwvzBYj+PJ09wYueK00RE5+i9Ff8DrjtVSXsziQv
SjJuUlv0kVvM8r3th4zBBNRhA4cinwqxhgqO4G+r2r9Gv0M2nKKOnWmyF+MSIRnh
gONOQZe5a7kQxKVWkLicS2IGUpPeQyTWaqZzYXsD+Dm6DXD57vYTURtUkwO0CDON
zT5XiS1HG1MZrw+V/Jai4HAvpF5WkTJXPc1Lv75BxJj3wOAw4MzEWCCdr/N/dt5/
+ULpEaSQfIg4L4iEj6rvabQyN0KbOxIDx+pPQ81izfj36wIrDqhyCNIdmVH/yARl
tkL4XDEl/pt7Y3t6jqFhy057lektowClWcPeq3DoL0LFYnjNPpYvIjRIAXdhaYiA
u2ViF8WdGzQ5tFeI7u3PQUG5NcPe+WOPOru3wMMrUhLgLHkCdNkjivP79qIPSTkC
GYkCHAQQAQgABgUCVvA48gAKCRC3hu8lqKOJoLRMEACmlyePsyE5CH7JALOWPDjT
f+ERbn+JUTKF+QS0XyWclA/BIK8qmGWfgH38T9nocFnkw17D3GP8msv8ll+T4TzW
9Kz9+GCUJcHzdsWj99npyeqG5tw+VfJctIBjsnX3mf4N0idvNrkAG5olbpR5UdsY
Yz62HstLqxibOg4zWhTyYvO6CjnszZrRJk0TYZON4cXN14WYq2OTrMaElx0My8o1
qVBnK58pIRzv72PmvQqUk5ZjhUyp9gxjqqCJDz0hVK61ZuGP6iKK8KCLTfSxeat0
5LAbz8aC58qlg5DVktevHOjBgnTa8B7BgJ7bQ9PLMa3lF4H1eSiR9+8ecpzEfGHI
LoeIDIYH7z7J/S0mTgV3u5brOMYO+mE9CEfps85tVVoyJrIR8mGEdtE2YmdQpdFz
YIYvRfq9tnXZjVsAAsC20Smw0LnjhYzAt9QJwZ9pFMXUTg6lC5xT+6LNrEY+JR3w
C16q36bcbCNj0cBv1A3x6OI5OQfpexhLPDgoDiI+qozJIdj8MzJ8W6KU1Z3yb3dq
ACk77yv37rGO6uduSHnSti26c/cUIy6XZBbXBdobE9O3tr8hwvTQ1FXBmYnBrdiz
U6tgxEA5czRC9HOkdk6y6ocbjmONpF6MxkpJAvTMk7IqC2/hisbV9x4utla+7tmN
ZU137QGcaK2AGQablVAy4YkCHAQQAQgABgUCVvCMigAKCRCkhaDtUbi3xAU7D/9g
UPZSJ8pbZV9TLaKD57Bc7B78HNV/B438ib4dI33iihMTBHnCB1giPE9X54QoV8AS
xrO/xveS1kkj78jERqUcED6ZHhMLb9SWs6CxUKdMdgovnIlFUc+t05D5mb6STi+z
NihwO0JI+n79qhETy73WLpC7RR0aMx7zYcbqp3NWPptcf1kVGJZGx+QbEHfVye98
T5pkH5Wp+7LSlup6AldQT/oifxdGxLXbECTnwozRvyMpAaphoEHrET1YOmKnmw/J
yi6DLpTb3XvSf5Tntzr7HklCEcL9FvYCoHxiXWawLhuPhSyrFYeYtF1ypmzTgaJW
yuTZ8sN9J+y7Tbchk/I6FpX+3YoTgPCcC7hv1Krs803N/3KuyBEvhzg7NYRikzO3
fxXlBG0RMm+662E7KlERU24izbWhGiYwl34+MaxrIO4oDvF79LEN7y0+SjL4V0B9
689d+HI1ZfS9O1xkOlW6y0QyagOzsTOUF12s2mWydFmipbYnIwsSsu6Nzk3yO4M+
qYABJXJ3tIFQPTd7xqmPNlJ8mFtmzHDhb3Pv6sRNFLLujYM9cJpuNMbAHWdohz1b
jBT9pZQ3zWpll5wotUvGmJd6hTAXdUgmZ7lh7Uq6axClMmiLe1WYntcNpb04PyyE
m2+GU5x123UTiSX2LGKa4t+HNSM8nJL8BJiGk80xVIkCHAQQAQoABgUCVa0OAwAK
CRDDvTXkbdRdpVR+D/4/37e8WqKOHNPteQu42sj0ZOfcqyVMA9TQ578F0s9MwoQu
qfVhXGSWevOctuMv2qTBjBfFjkdPrKR5L4LNAgMsu1epHU0DPcRZUCbh1P7Gpolm
Z8KgnjT5Wpl1AcuOCaP08VMrt/e/JndTHp6btn6HsLVtryNhlL7oaeYbDr6/ovHN
GHVIVSZgGP9f4Y8FiDpyfKav71vYLBMxtzM7lc3eFT1S10XhSW6k+8S5XldYWkLD
riRXDE85C+9QndpOoQaIICp3ye3JVnUxa1qhvsYj9uPt1M6hKiBSoXdplrB+hQc+
nqLNN3jxpGdmGmwrjtjqMhocMIguEqgARJOek3XKOppEhu+IcnJgU4edARJNLsBa
uiVBWY/6mZOFlZq6H48tVyziS2n/oIpi+aCc/fQeGs9zMTtFUohPfYtTcy9PecXM
OYpSu4p4tQ07oucnxfBkRUgTdM5VwX7YwTcRwp9XhHACUEGBhrwMH8Iz+sK2jLF3
FhJGkef1vFs0vqSf4I8DBFkYAKF848YyEcGHeINQloi3v0Kr2PpBxlRh+GPWwi++
QPKXQFzlTiyVtMzoo/lpmAWUJwj0dbAbH/mohtvWtA1WPHC2JRZ52JLThhpDrK3t
//Jdt2WHE91cMx7/2B0PK4O8/j7UVlsOJXpVPsGX5SFCeTB/iS4JtIwWN275zIkC
MwQQAQgAHRYhBFnKni0qMx3iUaokJ18Dx2fCR6TVBQJZDvZCAAoJEF8Dx2fCR6TV
oGkQAIjqaQ7tpdhDJ6ORNtLIt0TsWg0jg2rpoq+9Au36+UYBMuBJ3Py/tAsZ3cqQ
lig7lJiQqOuQZkbg1vcY4Kdad7AGa8Kq3sLn8h2XUlNU90X0KAwdCTA/YXxODlfU
CD2hl4vJEoH/FZtfUsaLNHLmz0brKGrWvChq00j5bPfp90KYKqamGb3a4/LG4DHL
4lmEBtP++YA0YqUQ3laOvKune2YwSGe4nKRarZnFiIn2OnH9w0vKN/x9IMGEtc5M
bQVgGtmT5km3DUuXMDforshue6c7ao4nMOC96ajkWYZhybqHJgLOrEGPVUkOaEe7
s1kx4ye9Ph3w/LXEE8Y8VFiZorkA/8PTtx0M9hrCVkDp0w8YTzFJ9DFutrImuPT6
+mNIk+0NQeuDsv492m/JXGLw/LRl97TmHpKME+vDd5NBLo4OShlDKHwPszYcpSJT
G9+5++csR95al3tWnuGX9V0/dO1s7Mv0f/z07nLB/tL+hEpqqA5aRiGzdx/KOrPZ
uhCTyfA3b2wvOblwf4A/E1yO7uzPTuSWnx1E14iZuaCPyZPXEh3XSYCLEnQ05jy5
0uGXCDVR+xiE/5i/L3IxyhJk6zn5GOW5b8Taq5s/dFS3zWiFS6l0zQ1VQmJH8jdG
LoBFvdVLZoAa1bihLo+nJVPR2RauWnxWoWk1NQoT3l02Lk6DiQI4BBMBAgAiBQJV
qUAEAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRBht7Um2Y8DU1CqD/9G
vr9Xu4uqsjDHRQWSfI0lqxElmFSRjF0awsPXzM7Q1rxV7dCxik4LeiOmpoVTOmqb
oo2/x5d938q7uPdYav2Q+RuNk2CG/LpXku9rgmTE7oszEqQliqKoXajUZ91rw19w
rTwYXLgLQvzM3CUAO+Z0yjjfza2Yc0ZtNN+3sF5VpGsT3Fb14aYZDaNg6yPFvkyx
p0B1lS4rwgL3lkeVQNHeAf0qqF9tBankGj3bgqK/5/YlTM2usb3x46bVBvwX2t4/
NnYM5hEnI57inwamX6SiMJc2e2QmBzAnVrXJETrDL1HOl4GUJ6hC4tL3Yw2d7515
BlSyRNkWhhdRp1/q9t1+ovSe48Ip2X2WF5/VA3ATfQhHKa3p+EkIV98VCMZ14x9K
IIeBwjyJyFBuvOEEIYZHdsAdqf1zYRtD6m6obcBrRiNfoNsYmNY4joDrVupI96ks
IxVpepXaZkQhplZ1mQ4eOdGtToIl1cb/4PibVgFnBgzrR4mQ27h4wzAwWdGweJZ/
tuGoqm3C6TwfIganajiPyKqsVFUkRsr9y12EDcfUCUq6D182t/AJ+qE0JIGO73tX
TdTbqPTgkyf2etnZQQZum3L7w41NvfxZfn+gLrUGDBXwqLjovDJvt8iZTPPyMTze
mOHuzf40Iq+9sf5V9PXZ/5X9+ymE3cTAbAk9MLd9fbkCDQRVqUD0ARAAr/Prvt+m
hVSPjNDPSDrTBVZ/7XLaUZvyIVggKa+snJoStrlJGTKKFgDVaYTOE3hP/+0fDdQh
97rjr4aRjd4hBbaNj0MzZdoSWYw3yT+/nidufmgPus0TIJMVO8I6rl3vgcfW/D3o
vNrLW/LjkTuM9a+p+D1J7woCfMSWiFMmOLPKFT7RBuY8edCVjyA6RP9K9Gj1sURS
eqNaHR9Gr4rW10s+FwUHWxxzbmIWqH0gApQYO6vyND5IMcKOBCWQU6Detuq1pQ6d
Uc+iF+sEz3Rk3C6d4WBBjtkVJSJ0KKan8Q3gJefOCMNhdRQDjZLwbzr4bgoAkLba
BFCjiZxWZ6HAdMfSCV8uZQrtMS7b0DUpY0vdH9Htl3JqOOkK9RorYDQBuPdkTYFI
NsmtWVsFV/LmR891mOF3fBRaoVoMeJVwiZyNlFY+dyWWFzLp+GoTLcQtmuR7OkmO
cBGxWSKPcZfPqhf4dVQud7bDR2RNfJ1Hqa5kj8Z422sseYDwHf/T9OWWYvLwKGZh
lUgpnzO3WCGrd/6EVNeC1mKXt4F7BmADov4Rdcrp1mPXiVt7oIxLaS6eBNf2y1TW
zjYj5ZFuKqIukDEJfqpwsE5asnCw56nae+7luGs8em1J9GEXhWzXG15UVyQJaFwu
B1iL8l7VcEQz4ABVrSTUWLLAKDsyqUbq2gsAEQEAAYkERAQYAQIADwUCValA9AIb
AgUJA8JnAAIpCRBht7Um2Y8DU8FdIAQZAQIABgUCValA9AAKCRAcacTlXpkF2y/F
D/oDrZm143Rv9NV9InnVJ0brpqbB7aulFfhR1LDuJ/GjeqGAQgJCZdHlzT2pfCXX
swUlYzcWEatvGcDkoaB5Ya2qs+6nhBk8pT6XYRrZAtIlKIGrlCqoSBm9HXguGv+E
IaEECr2z/Funx9so0mP+5aJn65M9u3lPmuAonj6DcHoM07WsfsXvQ4ut3fabFmzi
lLGeAdEDKIw8Hn3JBUOxUyFrQlOoL4/3qK1TO+cidz/2bATQQyIG2kNOSgHBslU+
e6/7sWOQ4ufmzm7dEsf197zPXGdXR88LT+d2uU2K4GkCffNUKxZqy9bXxXPwr4JB
jxLDQnDvl50GAWjPZAwXEd8Okwl5+8xp0HuZ217WUqT8ib0oUUfwh2H1vrMPRr/4
6i6O6THpCkV8BWF7axPYIibaeYwC4BkjZwK3tIL5ESf2f0xK4hbE3xhMTeqABQHo
Xd5rQ7SEaUuX7PlQ59fRs0Cz55vH8/o9zMm0PN6qmZFvRBeqjnklZcu+ZdP9+CMX
t81NMuzIK1X7EfpkUoam8YkYkwcCkRvPZrSHLXZFkfnx4jW543dPOfycjnv6hhKy
oXD9CBx0ZcOicsYmw9XMilBGD3b8ZdK6RYX4ywKNU6KUdFJjXB88+Ynv6QxDit1e
mMCHA1glzV9/k36iYLEIqgWBiwJeUUIcUqzgnBFtN13cyS6oEACUGUiPKbw3IkgG
W19ZyS6FBNfgGIGW0Y82Br0KlCyaXnX0R4+4u2h7kfR9NSnhRhsvRnPIkiZATa7D
+Ew1nfpsDTnti0c6g/gVw9TC/rCyXkkLztRHVcWEBdvnFJTSp2LeFaHSGbvvZfoI
GUzyUzoa1P98NmRIY1cxBoizVf8729/zAaD4fAslxoK/JsjjDvDUrRHtaNZmUle6
0Jl/yFFzR3zxb+pJliigoP2rZLt+ipomHJIhoXXWwfkRO9U/egJ8ZUhWEpZvROna
Nc9eVct5EBADxL7gHWjlceIz4ndI1eE9AdEZDdUZwOfjmK2DcXjFBfZC+jhJXjY0
xh3pPKQz90h9DIkM5WDcJPf6ep+MKSd/3hI2/JmmscQ+alwN6x6g8zDySMo3APA9
cUvEFGe0+CepVcNw03jU4faSrHiMXsUuVGbA2kHaYVUfzF5W5GbuHZZlGxoSiq+K
+HNG0RJUDa6bkSDvrcJVNw1iUrowP+LLwnNsy5kGuU4evnwcoN1w7LVbTPaq4RIa
iqvAD33kiA9q//UNKnK4k81z+hRNaWGliyGpgqh+V7MDIqPfT5TMLdH+ZjTeuLrN
S8KBcc2BmUpSwzdUReTqHmgO5peeIcsvO7GNMFWsgucZiAdIVE/zQv+SfP6jhS+r
jCPs0eeu5zl8/V+gXFE2wy3jTJEl9bkCDQRZS9m1ARAAvh1Nh4GgjpTFZy7uQRFz
5PPXdZTBI+Y4hTpF2heoFzZDI6SLyz64Ooglum3ZglQ9ac+ChTSsO36aw4b22kCM
9WDmkcl7wf21fG9o8gJDVjFjDWbwTWREaKjgS6s/Yb8f9gje/BGySojxynTi3zyT
UN94q9dhVjfiQ79UzXZdN9FyyIx2YO5tOo09hTWSZg16oxP47Mj1ATaS6UIrQMcM
nOp0kuc6SufXPSWsUA+g2lW0dmHgPvIHwUfcjWqT2elF01e9KOFe7im29G6zOS2M
Rx8cr6KRg/eNWpHh5aI4quRUhYk4Kw4ohQTbs9ed0YttS4PMK+sq6xHpb28X6Zgr
WnelPY9hfwcR4m7Ot3VQUG8JY9/aTlFCoeTgkhop+MCUI+dJeY8depIa0PTzdEmE
WRvPhTTv+CUdZ6v4z5LD6FhP+/5c6FCbcIb89Rp5fa53oYV5/KZf+0DUVgmpXFU7
J7ZrGgDeU7vIzmwr8kcx0vtsVm1dVwYLACpTaaQPbISQUDM8sEcqKAqD7hWKaxNs
b2M85L6q2/rnHq4g46yJzdR3b8EH+V9u+mUi9DIljDwcpvw7ReRQ9wPdDWLynngl
IeGImbjYfr324yaIl4vNORAkbsoCkS/qc5v6MvKvYNle5fzb9S9kCbNZmD9c5/bH
Pjj9ENeQvzrl2pFh6dc1o5cAEQEAAYkEcgQYAQgAJhYhBBTyZoLQkWzdgeN7bWG3
tSbZjwNTBQJZS9m1AhsCBQkDwmcAAkAJEGG3tSbZjwNTwXQgBBkBCAAdFiEE3OrF
2WE1uRxOpnKru769uyTG81UFAllL2bUACgkQu769uyTG81UFUw//bW5T7w2k8ukG
fpIcm0gB98VgxKenSCmU6N+Ii0DwcNtzW+pmVWl2TbHIXDpvuD69ODWBDMXu6gBk
rVzNEsK3uhzGe0tWA+5I7Vke3iEkbll7VRQlIOrw+n5NMvjeuDqKsMt1gMEEdgRK
ddYApEAi49vV7XnqkB2lLKfAnf6o/KqPm8MuQ+u0xYanupZCldwdpcx5rybj79Es
0iO9Gh/+3qOtR6ubOz3Vn78Lc3y6AP9pmtdOI2QX8foGK4hNmgHSP6uPLh/ERC9N
ir0Lc2hoEhHEkQ8CnEaccp70r03VkEQuMJQJPUyRsGZ/gIm0SAm9JJxWHXJk2/5N
UN83pHAX0LA4zxtWs4fVW5f8v9eIhFFPTZ4au+/cS9D4GFx4mlY34awcpAzrny2t
ntGEejY9HSJv4PuFZCmtyS2q61N9EU8yuBwVM9cp5HntzG+OT4HYugtI6ibehM0S
1Roy4ETwT+Ns41ffhCwdYMp8tzdeksQ35s7rkB9OJHj+q2dkGaV0FQb3FutbSpxb
P4zk/dLqyxuivdUPHGtf4W/qklxzCWBg0VDFA7PwatmEXRxTjx77RelTY0V7K54d
DyVv3Jh2+FzuaQZzzuIhv4gtqHntaqLnYl3h/QNLbOTE3ppvn9RUSR983Bd+M3Qh
bbwZrgG1m+hdUZUmji+wbK0wV0xHNEH+4BAAjbVzdNOs7hMvjY1wVDRFjvICVorN
dNdU3ELy/9BAoiwOs2+zjDXmsX+3YtdzwKvdpQ24O0TvH4Vo3BkvKkJ75EU7LroA
bYQ2423m1MY3eaBslmX7TUJ3XE+k7OZF8AmcftgP4nhC4IQSCtoBc9+ncyGN4da1
BpYO7b19tO0/HST8GHSrEcU9bGGdimS2eNkSgybA8wF6K0K9yvrpTNSZ7OBVlzQf
En8s70Gyzs/d6C/rTA+defnv3AMaciuINSEdFyfYq4wjt5PikvgceMAAkH/z69xT
Ng+6q3FQt/lyK7xX5qPMe2oFyDA1H+Cb/uL7ioo+jXh9gF+0fk8OP2IPzxYhBful
pVtgclmOuaekzaKeIv8NFW7GoA9OghziExePxg95OpL/VyQ7PJiAUj1pFovFk5HS
6ejVZNEGJ/A5zLc1PBIcr/phu0luqhXAhImsZS6858GWQllWULNWw8bX5Blo8Avc
fFVdq9iAK7aHN7g45ZR7Ze6qKHDyFv4XWuE/rj9C2mM/GAstvU0gGmbo6B1mNGMJ
uX3Gd3dG8fqFjE77OB2feJyfZ8UeF1nvG1hxlmuD1A5e6/osO9V7kjhXKzM2zSO1
1zHQ/5PlUisoUBjJ/QIK4v9RBNGtbRKso5X9Fke692lVgrdggDJ3j2QqMuTo71rA
VDLtxerc+GNq0GK5Ag0EXPA56gEQAK3x5otbuNfefm1BD4gJ4Y4EhVvMCdeUf1uN
1OqiWUz6KLCR2UE00QaS7v65D11Oh96bpxtJRe6HmQk0vLm1QgbigZku+kCMt0c/
zOyWBOCVDKahZJu/ayzimrzGgNunoTnV2SjCTIVK4QtkowUeA+Ilt9hfFCZvASIW
Eazj7bWXC0ji1U73OzVdvMkP1e2ibGRxhp494HGkj0metYhQq3vQk9mL9BzE8wba
yL9iK2cJVqZxXDxauL5bswAlbdLI2MlSrH9wg6Jvy91lXG91ZEhmZ2RaQU7q+/gi
i0QEJ+W3pwPQQjLciOfecMN/n2/vCl1ZeJ2PNWMoXQXyNSJfpL9m4TPiW92i2kKA
kaEOZ2LilUJhuZKEpt9+BxIn8FafjAI4co8D5cHvnPM+PXt1xZHwP0Yz+yC6wnLZ
9scNw50AM/Kcpbc6TUIQ0kPHsNMPCbZ89Ey63i61IvUUxKXvMENFptw+vWfD7MRG
OZKObdH3VrKtD1QqU//g/xeSGNzcUVatFDVDv9KWRSatOpKi5Bw6iUYxrYQ02Co1
45jugTnGc+zrhMsDND3Prq7R0dMw4MBLUDoLSXrous9VgahAsOQztZCfvPOcV2cp
Haew85SLSYx3Ksb/raIBG02+4XjF2xQCgIq2LfA6xdG5SGJd6Lg3ip2vwhogQgWB
gQoq1qrPABEBAAGJBHIEGAEKACYWIQQU8maC0JFs3YHje21ht7Um2Y8DUwUCXPA5
6gIbAgUJA8JnAAJACRBht7Um2Y8DU8F0IAQZAQoAHRYhBAl7MTB3rmKgL4TaTfGm
Zo+7fVcuBQJc8DnqAAoJEPGmZo+7fVcuilAQAIq/yZi3hoxzSgDblQlcxGKDaVck
Oo5kFRK8UWby3peB0bTMV3MWjiCqmN7ryQ3K3126G84yxXCTNSv7kOLsVxy+DBx5
TdfmeBamdX8ZxeDv2bm+KKlZR0FGjJn3SiLgPkrp+GWazOZbKIPsoOuN20h0aeMC
6mvO2Rz8kGUjCWa7/jQnINHkN/GnbnoOXOwn8cfwtt2Mkb8XRe+1lTBwd59ruYNe
0hazOwygmcSgvM6Rm8nXeuElxMdVAiYKXJLej34+ToNN/LgB3TD72cXStmn7z8Zd
kaKV+f81cyPR6Vi7laLIIGpquBGSsel5s0pb2PFFxrTZoNLZit/x2Wx47V30Vdk8
4FYjENEb75OpGEErwVRbTQOKREOutXlGwiJe2C1Gof6HCScTfeLybz8ooLn+lSXc
B/l/68wqxPezWYxfbMD42YtTEkJcvPvRaouhKEjxASP5EuT3K6T3tujsqg1K/rE3
Bei0krOnTxUCdU1f9B12ZXpntKehwSz5LpnQvr2wR98yBRoKQfT2Yi3XK7QNp46X
t8zaIPdVw7qMn3gODyiWSEGCtkRF7exdHZ9yIW2yofFhz3M/tjIvu89tRDwGiDER
8hBk8Z2mcbmlMzQ1Jf+aCQ5YjZUkUN7+s4VN1OT7yYpWcV1pR8cDjyoh5LRXTMwt
3g5mStiVbeIq9mmzogAQAJq6KK9Z28q8mAozKLd++JxefiovydXVfqWY4GoOfhtv
Q3Sa2mKpd+8b3nT7b2twIlKPqpIxWPxke7BWS2sShZfqWkltbAR1SH7fkOeN8Y5g
3yBbxvHQJ/KoTHnRb74MrlaP4v4MvuOAn+Xt0wrEt2OKOSD6/jIjnXqfNFuvHfHv
tTyf4fVX1sesN9JwRKevc+1ZLjN7jIOuv1en5grwDtGQhy0fZOjISLGKfy6299rn
p0alRqLuAhFn7Ru1ZjQVfqD9VEfDCEsjNOxoW95Aa7MVtdoaQ6FEBKK0q1tCjZ8P
f2oAyLGQo5y030D+cDd1lTwGUmPvbnz4/fqNAM6KWvZjtGVNJNlLlpd4A9vAgvAo
I1wONXSR56sZKqJXbOSIfS8AlzFWiSf6IyPpU5ozVHLGySAVqtrRW3ci4COeKsRL
85W4iSGCl5uYZcE9weotakUO0R0u/pi07BJwTR5hvn9IDdeyYreqDMQDMF9yHCXg
+oCGJAB/+1vHkSABRsWbCERyB33ExjnDr7z/kWOhfAdiZ1+MpP2IPtDEXcufKayS
gpBAZYEeLAmNoDwtg+milSD3B7TTM11IY1e/Utq70Az/8BOK6wfyIsmPfeh9CHoW
+JAbpbdOUs+rH+gjHE2Pls8BDLp1EoHVPGVavhfSlRbG6oF45s+gdL2PTKzEo+u7
=xFcH
-----END PGP PUBLIC KEY BLOCK-----

Smaller Firefox Updates

Back in 2014 I blogged about several ideas about how to make Firefox updates smaller.

Since then, we have been able to implement some of these ideas, and we also landed a few unexpected changes!

tl;dr

It's hard to measure exactly what the impact of all these changes are over time. As Firefox continues to evolve, new code and dependencies are added, old code removed, while at the same time the build system and installer/updater continue to see improvements. Nevertheless I was interested in comparing what the impact of all these changes would be.

To attempt a comparison, I've taken the latest release of Firefox as of March 6, 2019, which is Firefox 65.0.2. Since most of our users are on Windows, I've downloaded the win64 installer.

Next, I tried to reverse some of the changes made below. I re-compressed omni.ja, used bz2 compression for the MAR files, re-added the deleted images and startup cache, and used the old version of mbsdiff to generate the partial updates.

Format Current Size "Old" Size Improvement (%)
Installer 45,693,888 56,725,712 19%
Complete Update 49,410,488 70,366,869 30%
Partial Update (from 64.0.2) 14,935,692 28,080,719 47%

Small updates FTW!

Ideally most of our users are getting partial updates from version to version, and a nearly 50% reduction in partial update size is quite significant! Smaller updates mean users can update more quickly and reliably!

One of the largest contributors to our partial update sizes right now are the binary diff size for compiled code. For example, the patch for xul.dll alone is 13.8MB of the 14.9MB partial update right now. Diffing algorithms like courgette could help here, as could investigations into making our PGO process more deterministic.

Here are some of the things we've done to reduce update sizes in Firefox.

Shipping uncompressed omni.ja files

This one is a bit counter-intuitive. omni.ja files are basically just zip files, and originally were shipped as regular compressed zips. The zip format compressed each file in the archive independently, in contrast to something like .tar.bz2 where the entire archive is compressed at once. Having the individual files in the archive compressed makes both types of updates inefficient: complete updates are larger because compressing (in the MAR file) already compressed data (in the ZIP file) doesn't yield good results, and partial updates are larger because calculating a binary diff between two compressed blobs also doesn't yield good results. Also, our Windows installers have been using LZMA compression for a long time, and after switching to LZMA for update compression, we can achieve much greater compression ratios with LZMA of the raw data versus LZMA of zip (deflate) compressed data.

The expected impact of this change was ~10% smaller complete updates, ~40% smaller partial updates, and ~15% smaller installers for Windows 64 en-US builds.

Using LZMA compression for updates

Pretty straightforward idea: LZMA does a better job of compression than bz2. We also looked at brotli and zstd for compression, but LZMA performs the best so far for updates, and we're willing to spend quite a bit of CPU time to compress updates for the benefit of faster downloads.

LZMA compressed updates were first shipped for Firefox 56.

The expected impact of this change was 20% reduction for Windows 64 en-US updates.

Disable startup cache generation

This came out of some investigation about why partial updates were so large. I remember digging into this in the Toronto office with Jeff Muizelaar, and we noticed that one of the largest contributors to partial update sizes were the startup cache files. The buildid was encoded into the header of startup cache files, which effectively changes the entire compressed file. It was unclear whether shipping these provided any benefit, and so we experimented with turning them off. Telemetry didn't show any impact to startup times, and so we stopped shipping the startup cache as of Firefox 55.

The expected impact of this change was about 25% for a Windows 64 en-US partial update.

Optimized bsdiff

Adam Gashlin was working on a new binary diffing tool called bsopt, meant to generate patch files compatible with bspatch. As part of this work, he discovered that a few changes to the current mbsdiff implementation could substantially reduce partial update sizes. This first landed in Firefox 61.

The expected impact of this change was around 4.5% for partial updates for Window 64 builds.

Removed unused theme images

We removed nearly 1MB of unused images from Firefox 55. This shrinks all complete updates and full installers by about 1MB.

Optimize png images

By using a tool called zopflipng, we were able to losslessly recompress PNG files in-tree, and reduce the total size of these files by 2.4MB, or about 25%.

Reduce duplicate files we ship

We removed a few hundred kilobytes of duplicate files from Firefox 52, and put in place a check to prevent further duplicates from being shipped. It's hard to measure the long term impact of this, but I'd like to think that we've kept bloat to a minimum!