As I mentioned in my previous post, in my new role at Shopify, I've been doing backend development in Ruby. Previously I had been working nearly exclusively with Python for over ten years, so I was a bit nervous about the move.

In addition to my regular work, I also tried to solve other types of problems using Ruby. Advent of code was a really great way to learn a new language.

After nearly two years in the new role, I'd like to share some of my experiences, hopefully as a way to help and encourage others who would like to learn a new language, or are worried about moving into a new role, but don't know some specific technology.

Early pains

The first few weeks with Ruby were pretty tough. Luckily, Ruby shares some similarities with Python that make it a bit more approachable at first glance:

• Dynamically typed, interpreted language
• Class / method syntax is similar

Method calls don't need ()

One of the first things that tripped me up was not understanding that using () to call a method is not required in Ruby.

def greet
  puts "Hello, world!"
end

greet
# => "Hello, world!"

In fact, () are optional when passing arguments as well!

def greet(name)
  puts "Hello, #{name}"
end

greet "Chris"
# => "Hello, Chris"

This can be used to build some very nice DSL features, resulting in code that is much more readable. e.g. Rail's delegate method

require "rails" # to get delegate
class Message
  def greet
    puts "Hello!"
  end
end

class Greeter
  delegate :greet, to: :message
  attr_accessor :message
  def initialize
    @message = Message.new
  end
end

Greeter.new.greet
# => "Hello!"

Implicit return

Like Rust, the result of the last expression in Ruby is used as the return value of a function.

def add_one(x)
  x + 1
end

add_one(2)
# => 3

Strings & Symbols

Ruby has a concept called symbols, which are a kind of identifier using the : prefix. They're often used to reference method names, since you can't get a reference to a method just by accessing it by name like in Python. E.g. obj.foo will call the foo method on obj in Ruby, whereas it will give you a reference to the foo method in Python. The equivalent in Ruby would be obj.method(:foo)

Symbols are also used for named parameters in method calls. E.g.

def greet(message:)
  puts "Hello #{message}"
end

greet(message: "world!")
# => "Hello world!"

However, where symbols presented me with the steepest learning curve is how they're handled in Hash (i.e. dict) literals.

a = "key"
h = {
  a: 1,
  a => 2,
}
# => { :a=>1, "key"=>2 }

It's extremely easy to get the two ways of defining a value mixed up. I've wasted an embarrassing number of hours on bugs caused by mixing up strings and symbols as the keys in hashes.

Range expressions

Ruby has nice built-in syntax for ranges:

(0..5).to_a
# => [0, 1, 2, 3, 4, 5]
(0...5).to_a
# => [0, 1, 2, 3, 4]

These are super convenient, but I almost always forget which form is inclusive and which is exclusive.

A-ha! moments

Blocks

Before really learning Ruby, I remember trying to read up on what blocks were...and not really getting it.

The best way I can come up with to explain them now is that they're a kind of anonymous function / closure.

Part of my confusion was not understanding that there are a few ways of defining and calling blocks.

These are equivalent:

mymap(0...5) do |x|
  x ** 2
end

# is the same as

mymap(0...5) { |x| x ** 2 }

The block's arguments are passed via the identifiers between the pipe (|) symbols.

In both cases, the mymap method is being passed a block, which in our case gets executed once per element (but that's completely up to the implementation of mymap). The block can be named as an explicit function parameter, and this could be written as:

def mymap(obj, &block)
  result = []
  for i in obj
    result.push(block.call(i))
  end
  result
end

The block can also be passed implicitly (check using the block_given? method) and called via yield:

def mymap(obj)
  result = []
  for i in obj
    result.push(yield i)
  end
  result
end

Once I wrapped my head around blocks, I found them to be very useful!

"&:" idiom

Ruby has this really neat shorthand for creating a block that calls a method on an object: "&:". It's used like this:

["hello", "world"].map(&:upcase)
# => ["HELLO", "WORLD"]

There are a few things going on here:

• :upcase is a Symbol referring to the upcase method on the object
• & tries to convert its argument to a kind of closure using the argument's .to_proc method. Symbol#to_proc returns a closure that calls the given method on the passed in object.

The net result is something equivalent to

["hello", "world"].map { |s| s.send(:upcase) }
# OR
["hello", "world"].map { |s| s.upcase }

Brian Storti explains this in much more detail in his blog post.

Enumeration

Ruby has fantastic enumeration primitives built in, just checkout the Enumerable module. Most of the basic container types in Ruby support Enumerable; when combined with blocks, this makes filtering and transforming data in Ruby a joy.

It also fits my brain better than Python's generator / comprehension expressions. When writing Python code to transform data, I often found that I was writing some code, then backing the cursor up to the beginning of the line.

In Ruby the data and logic flow from left to right, which makes it easy to chain thing together.

(0...5).map { |x| x ** 2 }

In Python the logic is on left but the data is on the right.

[x**2 for x in range(5)]

If I want to get only the square numbers that are even, I would simply add this in Ruby:

(0...5).map { |x| x ** 2 }.filter(&:even?)

Whereas in Python I would need to introduce more code before/after the expression I already have to achieve the same result:

[y for y in [x**2 for x in range(5)] if y % 2 == 0]

tl;dr

Ruby is a really nice language. I'm glad I've had the opportunity to learn it!

Python has a philosophy of "There should be one-- and preferably only one --obvious way to do it.". I think this helps with Python's readability at the cost of expressibility, elegance, and conciseness in some cases.

In Ruby there are often multiple ways of achieving the same thing. It has a richer vocabulary for expressing ideas in code. This richness allows for more elegant code in many cases, although this perhaps requires more effort on the part of the reader to understand the code.

Recently I've been digging into some performance optimizations with some code that transfers dozens of assets into a bucket in Google Cloud Storage (GCS)

The code in question is using Google's Ruby gem, and uploads a set of files in sequence, something like this:

require "google/cloud/storage"
storage = Google::Cloud::Storage.new(...)
bucket = storage.bucket(...)

files.each do |filename|
  bucket.create_file(filename, filename)
end

For this naive implementation, it takes about 45 seconds to upload 75 files, totalling about 350k of data. That works out to about 600ms per file! This seemed awfully slow, so I wanted to understand what was going on.

(note that all the timings here were collected on my laptop running on my home wifi)

Jump down to the TL;DR for spoilers, or keep reading for way too much detail.

Simplify the dataset

First, I wanted to see if the size of the files had any impact on the speed here, so I created a test with just 50 empty text files:

$ touch assets/{1..50}.txt
$ time ruby gcs.rb !$
ruby gcs.rb assets/{1..50}.txt  0.61s user 0.20s system 2% cpu 27.951 total

That's still about 28 seconds, or about 550ms per asset, which seems super slow for a set of empty files.

Start digging

Looking at the http traffic generated by running this code, it appears as though Google's gem is creating two(!!) new TCP / https connections per asset. The default behaviour of the gem is to use a "resumeable upload", where one connection is issued to start the upload, and then a second connection to transfer the data and finalize the upload. It doesn't appear as though any connection pooling is happening either.

It looks like there's room for some improvement.

A few ideas came to mind:

• Use a single request per asset
• Use connection pooling
• Run requests in parallel (using threading or async)
• Use HTTP/2

Single request per asset

The GCS API is pretty simple, so it's straightforward to implement a basic upload function in Ruby.

def upload_object(bucket, filename)
  Net::HTTP.post(
    URI("https://storage.googleapis.com/upload/storage/v1/b/#{bucket}/o?name=#{filename}"),
    File.read(filename),
    {"Authorization" => "Bearer #{ENV["AUTH"]}"}
  )
end

This uses one connection per asset, and brings the time down to about 11 seconds for our 50 empty assets, which is less than half of the naive version.

Re-using the same connection

Net::HTTP supports re-using the same connection, we just need to restructure the code a little bit:

def upload_object(client, bucket, filename)
  uri = URI("https://storage.googleapis.com/upload/storage/v1/b/#{bucket}/o?name=#{filename}")
  req = Net::HTTP::Post.new(uri)
  req["Authorization"] = "Bearer #{ENV["AUTH"]}"
  req.body = File.read(filename)
  client.request(req)
end

Net::HTTP.start("storage.googleapis.com", 443, use_ssl: true) do |http|
  files.each do |filename|
    upload_object(http, bucket, filename)
  end
end

This runs a bit faster now, in 8 seconds total.

Parallelization

Ruby has a few concurrency models we can play with. I try to avoid threading wherever possible, and use async libraries. Luckily, Ruby's async gem handles wrapping Net::HTTP quite well:

Async do
  barrier = Async::Barrier.new
  files.each do |filename|
    barrier.async do 
      resp = upload_object(bucket, filename)
    end
  end
  barrier.wait
end

Now we can finish all 50 uploads in about 1.3s

HTTP/2

There are various options for making HTTP2 requests in Ruby. One we can use is async-http, which integrates well with the async gem used above. Another gem that's worked well for me is httpx.

Async do
  barrier = Async::Barrier.new
  internet = Async::HTTP::Internet.new

  files.each do |filename|
    barrier.async do
      resp = internet.post(
        "https://storage.googleapis.com/upload/storage/v1/b/#{bucket}/o?name=#{filename}",
        {"Authorization" => "Bearer #{ENV["AUTH"]}"},
        File.read(filename))
    end
  end
  barrier.wait
end

This finishes all 50 requests in 0.4s!

Looking back at our initial data set which took 45 seconds to run, we can now do in 0.9 seconds.

TL;DR

To summarize, here are the times for uploading 50 empty files to a bucket:

I'm really shocked at how much faster HTTP/2 here is.

Consumers of Google Cloud APIs should take a look at the libraries that they're using, and see if they can switch over to ones that support HTTP/2. I'm curious if other client libraries for the Google Cloud APIs support HTTP/2.

Can the Ruby gem support HTTP/2? There's an open issue on the github repo to switch the underlying client implementation to Faraday, which would allow one to use an HTTP/2 aware client under the hood. I've started working on a draft PR to see what would be involved in switching, but there are some major barriers at this point.