chris' random ramblings

How to fix Let's Encrypt / acme.sh renewal on Synology

chris — Sun, 27 Sep 2026 04:00:00 GMT

I use a Let's Encrypt certificate so Synology Photos can connect to my NAS over HTTPS without a certificate warning. I renew it with acme.sh using a DNS challenge.

When the certificate expired, Photos stopped syncing and renewal failed with:

curl: (60) SSL certificate problem: certificate has expired

The error made it look like acme.sh didn't trust a certificate authority. The actual problem was that the DNS-over-HTTPS (DoH) request was reaching my NAS instead of Cloudflare.

Finding the wrong certificate

During DNS validation, the renewal script queried cloudflare-dns.com and dns.google over DNS-over-HTTPS. I checked the certificate returned for Cloudflare:

openssl s_client \
  -connect cloudflare-dns.com:443 \
  -servername cloudflare-dns.com \
  -showcerts </dev/null 2>/dev/null |
openssl x509 -noout -subject -issuer -dates

The subject was my NAS's hostname. A local lookup showed that cloudflare-dns.com resolved to 0.0.0.0 and :::

nslookup cloudflare-dns.com

My router uses a NextDNS profile with Block Bypass Methods enabled. It was blocking the DNS-over-HTTPS providers and returning those addresses. The connection went back to the NAS, which presented its expired certificate. That's why curl complained about an expired certificate while acme.sh was trying to contact Cloudflare.

I checked the same name using Cloudflare's resolver directly:

nslookup cloudflare-dns.com 1.1.1.1

That returned the expected address. I added cloudflare-dns.com and dns.google to the NextDNS allowlist.

Clearing the failed challenge and renewing

The first attempt had created the ACME DNS TXT record before it failed. After allowing the DNS-over-HTTPS providers, the next attempt stopped with Cloudflare error 81058 because that TXT record already existed. I deleted the stale _acme-challenge record from Cloudflare, checked that it was gone, and ran the renewal again. I also upgraded acme.sh from 3.0.9 to 3.1.5. The script deployed the new certificate into DSM with sudo.

The fix for this error was the NextDNS allowlist. The certificate subject gave away the real problem: I was seeing my NAS's certificate in a connection that should have gone to Cloudflare.

Separately: adding CA roots to DSM

I also updated DSM's CA roots, but that was separate maintenance, not the fix above. On DSM 7, custom root certificates go in this directory as .crt files:

/usr/syno/etc/security-profile/ca-bundle-profile/ca-certificates/

With SSH enabled, I copied the current ISRG Root X1 and X2 certificates from Let's Encrypt into that directory and rebuilt DSM's trust bundle:

sudo mkdir -p /usr/syno/etc/security-profile/ca-bundle-profile/ca-certificates
sudo cp isrgrootx1.pem /usr/syno/etc/security-profile/ca-bundle-profile/ca-certificates/isrg-root-x1.crt
sudo cp isrg-root-x2.pem /usr/syno/etc/security-profile/ca-bundle-profile/ca-certificates/isrg-root-x2.crt
sudo chmod 644 /usr/syno/etc/security-profile/ca-bundle-profile/ca-certificates/*.crt
sudo /usr/syno/bin/update-ca-certificates.sh

This updates roots DSM uses to verify certificates from other services. It doesn't change the certificate DSM presents to Photos clients; that is managed under Control Panel → Security → Certificate. Synology's certificate guide covers that separate step.

Making a Vim watch face glow

chris — Sun, 27 Sep 2026 04:00:00 GMT

I wanted a Vim styled watch face for my Garmin Fenix 8. Line numbers, a filename at the top, and status information along the bottom. The time and other watch data could be the contents of the file.

I also wanted it to look like an old green CRT, with glowing letters and scanlines across the display. Getting the text onto the screen was straightforward. Getting it to glow took quite a few attempts. Here's where I ended up with Vim Glow:

Getting the clock to glow

My first approach was to build bitmap font atlases from Berkeley Mono using BMFont. An atlas puts the characters into one image; the watch draws the part of the image containing the character it needs.

The plan was to layer blurred copies behind the sharp green letters. I tried fonts at different sizes and overlapping passes to get a soft halo around the text. That meant loading several large atlases. The first version loaded nine and crashed when the simulator ran out of graphics memory.

Smaller atlases helped with memory, but I still couldn't get the glow to blend correctly. The blur was visible in the generated PNGs, yet the clock looked sharp in the simulator. Drawing the blurred atlas on its own produced a black rectangle around it. Other transparent glyph images disappeared entirely. I tried blend modes and a bitmap buffer with alpha blending enabled, but couldn't get transparency working reliably.

Eventually I moved the clock's glow into the images themselves. I generated a separate image for each digit and the colon, with the blur and sharp letter already combined against a black background. The watch only has to draw the finished image.

The build script crops each character out of the atlas, then uses ImageMagick to blur its mask:

execFileSync("magick", [
  sharpPath,
  "-alpha", "extract",
  "-blur", `0x${blurRadius}`,
  "-evaluate", "multiply", glowAlpha,
  maskPath,
]);

It colours that blurred mask green and combines it with the sharp character on black. All the images use the same cell width, so the clock keeps its monospace spacing. Finally, the digits had the glow I'd been trying to get.

Putting the scanlines back

The scanlines had been troublesome for the same reason. I'd been drawing them behind the text, expecting them to show through the transparent parts of the letters. The opaque clock images now painted black rectangles over them. I had a glowing clock, but the scanlines stopped wherever a character was drawn.

The fix was to draw the scanlines last, over the letters and their glow. Multiply blending darkens the pixels already on screen:

function drawScanlines(dc, awake) {
    var color = awake ? SCANLINE_WAKE : SCANLINE;
    dc.setBlendMode(Graphics.BLEND_MODE_MULTIPLY);
    dc.setColor(color, BG);
    for (var y = 0; y < HEIGHT; y += 4) {
        dc.drawLine(12, y, WIDTH - 12, y);
    }
    dc.setBlendMode(Graphics.BLEND_MODE_DEFAULT);
}

Now the lines run across the whole face, including the clock. I'd originally expected to build the effect from transparent layers, but the opaque character images with scanlines drawn over them still give it the look I wanted.

Running the simulator locally

I ran Garmin's Connect IQ simulator in Docker. It needs older libraries, including GTK and WebKitGTK, and I wanted to keep those dependencies off my main system. The container runs the simulator window on my desktop, so I can load the watch face and see how the glow and scanlines look as I change them.

Try Vim Glow

If you want to try it on your Fenix 8, I made a web installer. Plug in the watch and open the page in Chrome or Edge on a desktop computer.

need: for when you just need simple build dependencies

chris — Sat, 26 Sep 2026 21:57:14 GMT

Sometimes I need to express a simple dependency between files. For example, if I have a directory full of images, I want to generate a thumbnail for each image, and only regenerate the thumbnails that are out of date.

just is great as a generic command runner. I frequently set up a justfile with common targets like test, format, and clean.

But just doesn't track file dependencies. I could write a recipe that loops over all the images:

thumbs:
    for f in images/*.jpg; do magick "$f" -thumbnail 200x200 "thumbs/${f#images/}"; done

clean:
    rm -rf thumbs

This regenerates every thumbnail each time I run just thumbs, even if most of them haven't changed.

Why not Make?

Make can track those file dependencies:

.PHONY: all-thumbs clean
all-thumbs: $(patsubst images/%.jpg,thumbs/%.jpg,$(wildcard images/*.jpg))

thumbs/%.jpg: images/%.jpg | thumbs
    magick $< -thumbnail 200x200 $@

thumbs:
    mkdir -p thumbs

clean:
    rm -rf thumbs

I've always found Make annoying for this kind of thing.

You have to mark command-like targets such as clean and all-thumbs as phony, or Make may mistake a file with that name for an up-to-date target.
You have to create the output directory yourself.
Recipes need literal tabs. I've lost far too much time because a Makefile was missing one.

Why you need `need`

I wrote need to address these annoyances. Need only handles file dependencies. It doesn't try to be a command runner.

In need, I express dependencies between files and give it a recipe for building the outputs from the inputs. need compares file contents, so changing an input makes its outputs stale even if the timestamps are misleading. It also creates output directories automatically.

So I keep my justfile for running commands and add a needfile to express file dependencies. The thumbnail example becomes:

# justfile
thumbs:
    need get -j 'thumbs/%.jpg: images/%.jpg' -- images/*.jpg

clean:
    need clean

# needfile
thumbs/%.jpg: images/%.jpg
  magick {{in}} -thumbnail 200x200 {{out}}

Here, the shell expands images/*.jpg into the input filenames. need get maps each images/name.jpg to thumbs/name.jpg, then builds those targets. The -j option lets independent thumbnails build in parallel. If nothing changed, Need has nothing to do.

Need can also publish outputs atomically, so a failed recipe doesn't leave a partially written thumbnail in place. Its dependency signatures use file contents, not timestamps alone. For a bigger example, it can use compiler depfiles to track headers discovered while compiling.

I still use just for commands like test and format. need handles the file artifacts those commands depend on.

need on GitHub

need-tool on crates.io

Snatching text from Herdr

chris — Sat, 26 Sep 2026 18:21:53 GMT

A couple of years ago I wrote about Extrakto, a tmux plugin that lets me find text in my terminal and insert it into the command I'm typing. I've missed that little trick while working in Herdr. A build prints the path I need, and there I am selecting it with the mouse or typing it out again.

So I built herdr-snatch. It puts an fzf picker over Herdr's output. Search for a path, URL, commit hash, word, or whole line, then press Enter to insert it into the pane you came from.

For example, if a build prints src/foo/parser.rs, I can type vim, open Snatch with prefix+Tab, search for pars, and press Enter. The path appears after vim, ready for me to run the command. Snatch doesn't press Enter for me, which is probably for the best.

It searches the visible output in the current tab by default. If the text was in another pane, or has scrolled away, I can switch to the whole workspace or retained scrollback without losing my search. Ctrl-Y copies the selection to the clipboard, and Ctrl-O opens a URL or file instead of inserting it.

To install it, you'll need Herdr 0.8.2 or newer, Python 3.9 or newer, and fzf:

herdr plugin install catlee/herdr-snatch

The README has the key binding I use and the other picker shortcuts. Right now this is a small tool for the paths and bits of output I keep reusing. If it misses something useful in your terminal, let me know!

CO2 Monitoring with ESPHome

chris — Sat, 22 Feb 2025 19:26:27 GMT

Help! My Head Hurts and My Nose is Burning: A CO2 Wake-Up Call 😵💨

For the past few years, I've been working from my small home office. This winter has been colder than usual, so I've been keeping the windows closed with little thought to the air around me. But then I started noticing a pattern — by the afternoon, headaches would creep in, and sometimes, upon reentering my office, I’d be hit with a strange, burning sensation in my nose. It smelled like iron, reminiscent of a bloody nose or the sharp tang of sticking my head into my fermentation chamber while brewing beer. Little did I know, these were all warning signs that something was off with my air quality.

Monitoring indoor air quality has become increasingly important, especially with the growing awareness of CO2 levels and their impact on health and productivity.

I'm a big fan of DIY solutions whenever possible, so I researched options available for hobbyist home electronics and found a great solution.

ESPHome: A Smart DIY Approach to Air Quality Monitoring 🔧

ESPHome is a powerful open-source framework that allows you to easily create custom firmware for ESP8266 and ESP32-based devices. It enables seamless integration with smart home platforms like Home Assistant and provides a user-friendly YAML-based configuration system. Whether you're a beginner or an experienced hobbyist, ESPHome simplifies the process of building and automating IoT devices.

Using an ESP8266 board, an SCD41 CO2 sensor, and a simple buzzer, I created a CO2 monitor for my office. This system integrates with Home Assistant and generates an audible alarm when CO2 levels rise too high.

Setting up ESPHome

Installation

Install ESPHome using the following command:

pip install esphome

Then create a configuration file for the sensor, e.g., co2.yaml. My configuration looks something like this:

esphome:
  name: co2

esp8266:
  board: nodemcuv2

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  fast_connect: true

web_server:
  port: 80
  ota: false

logger:
  hardware_uart: uart1

api:
  encryption:
    key: !secret api_key
  actions:
    - action: rtttl_play
      variables:
        song_str: string
      then:
        - rtttl.play:
            rtttl: !lambda 'return song_str;'

ota:
  platform: esphome
  password: !secret ota_password

i2c:
  sda: D1
  scl: D2

output:
  - platform: esp8266_pwm
    pin: D3
    id: rtttl_out

rtttl:
  output: rtttl_out

sensor:
  - platform: scd4x
    co2:
      name: "CO2"
      id: co2
    temperature:
      name: "Temperature"
    humidity:
      name: "Humidity"

globals:
  - id: last_co2
    type: int
    restore_value: no
    initial_value: '0'

time:
  - platform: sntp
    on_time:
      - seconds: 0
        then: 
        - if:
            condition:
              lambda: |-
                int v = id(co2).state;
                return (v > 1000 && v > id(last_co2));
            then:
              - rtttl.play: 'siren:d=8,o=5,b=100:d,e,d,e,d,e,d,e'
              - output.turn_on: buzzer
              - delay: 5s
              - output.turn_off: buzzer
        - globals.set:
            id: last_co2
            value: !lambda return id(co2).state;

Put your WiFi credentials and other sensitive information in secrets.yaml.

Wiring 🔌

Connect the components as follows:

SCD41 VCC → 3.3V
SCD41 GND → GND
SCD41 SDA → ESP8266 D1 (GPIO5)
SCD41 SCL → ESP8266 D2 (GPIO4)
Buzzer Positive → ESP8266 D3 (GPIO2)
Buzzer Negative → GND

Uploading the Code 🚀

Connect the ESP8266 to your computer via USB.
Run the following command to upload the firmware: sh esphome run co2.yaml
Once uploaded, the ESP8266 will connect to your Wi-Fi network and start transmitting CO2 data.

Integration with Home Assistant 🏡

If you have Home Assistant running, simply add the ESPHome integration, and the sensor should automatically appear in your dashboard.

Conclusion 🎉

With just a few components and ESPHome, you can easily build a smart CO2 monitor for your home or office. This setup allows for real-time monitoring and integration with home automation platforms, providing valuable insights into indoor air quality.

Building this project not only improved my awareness of indoor air quality but also gave me peace of mind knowing that I have an early warning system when CO2 levels rise too high. If you've ever experienced similar symptoms or want to improve your home’s air quality, I highly recommend giving this project a try!

Advent of Code - 2024

chris — Sun, 16 Feb 2025 00:20:34 GMT

Advent of Code 2024 in Zig

This year, I decided to make things interesting and tackle Advent of Code using Zig. Zig is a low-level systems language that aims to be simple, predictable, and efficient. Coming from a Ruby, Python, and Rust background, diving into Zig was a mix of fun, frustration, and some unexpected insights.

What I Liked

Iteration Feels Good 🚀

Zig has a few neat iteration features. For example, for loops allow you to specify an index parameter, which makes enumeration straightforward. Similarly, while loops integrate nicely with the error and optional types, reducing boilerplate. More on that below!

fn printNumbers() void {
    const numbers = [_]i32{1, 2, 3, 4, 5};
    for (numbers, 0..) |num, index| {
        std.debug.print("Index: {}, Value: {}\n", .{index, num});
    }
}

fn whileLoopExample() void {
    var i: i32 = 0;
    while (i < 5) : (i += 1) {
        std.debug.print("Iteration: {}, Value: {}\n", .{i, i * 2});
    }
}

fn iteratorExample(iter: anytype) void {
    // iter.next() returns an Optional value;
    // the loop will terminate when the value is `null`
    // if the value is present, then it's passed in as the captured
    // variable |val| here.
    while (iter.next()) |val| {
        std.debug.print("Value: {}\n", .{val});
    }
}

`else` Clauses for loops 🔁

A cool feature is the else clause for for and while loops. This lets you handle cases where a loop completes without breaking, which is handy for search algorithms and cleanup logic. Python also supports else clauses in for loops, while Ruby does not. This is something I miss about Python when working in Ruby. Python's else in loops allows for a clean way to handle cases where a loop completes without finding a match, while in Ruby, you often have to rely on find or detect instead.

fn findValue() void {
    const values = [_]i32{1, 2, 3, 4, 5};
    for (values) |v| {
        if (v == 10) {
            return;
        }
    } else {
        std.debug.print("Value not found!\n", .{});
    }
}

Option and Error Union Types ❓⚠️

Zig does not have exceptions, instead relying on option and error union types. I really like this style of programming, which you see much more commonly in Rust. Having to explicitly deal with errors rather than letting exceptions bubble up through the stack makes the code a lot easier to reason about. I like that Zig has built in support for working with options and errors.

Optionals (?T) help represent values that may be absent, and errors (!T) are returned instead of thrown.

You can use orelse to provide a fallback value for an optional expression, which makes handling possibly null values straightforward.

fn getNumber() ?i32 {
    return null;
}

fn example() void {
    const value = getNumber() orelse 42;
    std.debug.print("Number: {}", .{value});
}

Optionals can also be used in an if statement or while loop to conditionally handle the presence of a value:

fn checkOptional() void {
    const maybe_value = getNumber();
    if (maybe_value) |val| {
        std.debug.print("Got a value: {}", .{val});
    } else {
        std.debug.print("No value found!", .{});
    }
}

Similarly, catch is used for handling errors when working with error unions:

fn mightFail() !i32 {
    return error.Failure;
}

fn anotherFunction() !i32 {
    return 42;
}

fn handleErrors() void {
    const result = mightFail() catch |err| {
        std.debug.print("Error occurred: {}", .{err});
        return;
    };
    std.debug.print("Success: {}", .{result});
}

And try is used to unwrap the value from an error union, returning any errors to the caller. This is similar to Rust's ? operator.

fn handleErrorsWithTry() !void {
    // If anotherFunction() returns an error, we return it here
    const result = try anotherFunction();
    std.debug.print("Got result: {}", .{result});
}

Labelled Breaks ⛔

Nested loops are usually annoying, but Zig’s labelled break statements make them way easier to manage. No more tracking weird flags just to escape a loop early.

fn nestedLoop() void {
    outer: for (0..3) |i| {
        for (0..3) |j| {
            if (i == 1 and j == 1) {
                break :outer;
            }
            std.debug.print("{} {}\n", .{i, j});
        }
    }
}

Pointers (Yes, Actually) ➡️

Pointers generally strike fear into the heart of programmers, but in Zig they feel well-structured and easy to use. The explicit handling makes memory management clearer and less error-prone. In Rust, you have to fight against the borrow checker a lot, especially when working with tree structures—which are very common in Advent of Code! Being able to use pointers makes writing these kinds of data structures far less cumbersome... as long as you don't mind debugging a few segfaults.

fn pointerExample() void {
    var x: i32 = 42;
    const ptr: *i32 = &x;
    std.debug.print("Value: {}\n", .{ptr.*});
}

Memory Management 🏗️

Building on the advantages of pointers, I appreciated having more direct control over memory allocations. Zig’s explicit allocator model and the defer keyword make resource management predictable and efficient. Being able to choose and switch allocators as needed, especially leveraging an arena allocator when performance and cleanup efficiency matter, is a huge plus. It’s refreshing to have fine-grained memory control without excessive boilerplate.

Here's an example of using an arena allocator for efficient memory management:

const std = @import("std");

fn example(allocator: std.mem.Allocator) !void {
    var list = std.ArrayList(i32).init(allocator);
    defer list.deinit();

    try list.append(42);
    try list.append(7);

    for (list.items) |item| {
        std.debug.print("{}", .{item});
    }
}

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();

    try example(allocator);
}

Annoyances 😬

Zig isn't perfect (yet? 😛). A few things that frustrated me:

Verbose Debug Printing: Zig doesn’t have varargs, so you have to use struct literals when passing arguments to std.debug.print. While this makes sense, it can feel tedious to wrap arguments in an anonymous struct .{ ... } all the time.
Lack of Functional Programming Constructs: Coming from Ruby and Rust, I missed having functional constructs like map and filter. In Zig, accomplishing similar transformations often requires writing explicit loops, fighting with types, which made writing data transformation code more cumbersome.
Cryptic Compiler Errors: Some compiler errors, particularly related to comptime operations like string formatting, can be difficult to trace. The error messages don’t always point to the exact issue in the source code, which makes debugging trickier. Hopefully, this improves in future versions of Zig.

Final Thoughts

Despite the quirks, using Zig for Advent of Code was a fun challenge. It forced me to think differently about problem-solving and get my hands dirty with lower-level programming again. The explicit memory management and structured error handling are great, but I definitely missed the expressiveness of Ruby and Rust.

If you’re curious about my solutions, you can check them out on GitHub. Zig is an interesting language, and while I won’t be using it for everything, it’s definitely worth exploring!

Packing bits with Rust & Ruby

chris — Fri, 18 Oct 2024 01:24:39 GMT

The missing C of CHD

One element of the CHD (compress-hash-displace) algorithm that I didn't implement in my previous post was the "compress" part.

This algorithm generates an auxiliary table of seeds that are used to prevent hash collisions in the data set. These seeds need to be encoded somehow and transmitted along with the rest of the data in order to perform lookups later on. The number of seeds (called r in the algorithm) here is usually proportional to the number of elements in the input. Having a larger r means that it's easier to compute seeds that avoid collisions, and therefore faster to compute the perfect hash. Reducing r results in a more compact data structure at the expense of more compute up-front.

Packing seeds

Seeds are generally tried starting from 0, and typically don't end up being very large. Encoding these values as a basic array of 8/16/32-bit integers is a waste of space.

I wanted to improve on my implementation of efficient encoding of hashes by doing some simple bit packing of the seeds.

The basic idea is that for a set of integers, we find the maximum value, and therefore the maximum number of bits (b) needed to represent that value. We can then encode all the integers using b bits instead of a fixed number of bits.

There's a Rust crate bitpacking that does exactly this! And it runs super duper fast, assuming that you can arrange your data into groups of 32/128/256 integers. The API is really simple to use as well:

use bitpacking::{BitPacker, BitPacker4x};

fn main() {
    let data: Vec<u32> = (0..128).map(|i| i % 8).collect();
    let packer = BitPacker4x::new();
    let num_bits = packer.num_bits(&data);
    let mut compressed = vec![0u8; 4 * BitPacker4x::BLOCK_LEN];
    let len = packer.compress(&data, compressed.as_mut_slice(), num_bits);
    compressed.truncate(len);

    println!("Compressed data: {:?}", compressed);
}

Bridging the gap between Rust & Ruby

I wanted to use this from Ruby code though...time to bust out magnus!

Magnus is a crate which makes it really easy to write Ruby extensions using Rust. It takes care of most of the heavy lifting of converting to/from Ruby & Rust types.

#[magnus::wrap(class="BitPacking::BitPacker4x")]
struct BitPacker4x(bitpacking::BitPacker4x)

impl BitPacker4x {
  // ...
  fn compress(
      ruby: &Ruby,
      rb_self: &Self,
      decompressed: Vec<u32>,
      num_bits: u8,
  ) -> RString {
      let mut compressed = vec![0u8; 4 * Self::BLOCK_LEN];
      let len = rb_self.0 // refers to underlying bitpacking::BitPacker4x struct
          .compress(&decompressed, compressed.as_mut_slice(), num_bits);
      compressed.truncate(len);
      ruby.str_from_slice(compressed.as_slice())
  }
}

This lets me write Ruby code like this:

data = 128.times.map { |i| i % 8 }
packer = BitPacking::BitPacker4x.new
num_bits = packer.num_bits(data)
compressed = packer.compress(data, num_bits)

Here we have these 128 integers represented in 48 bytes, or 3 bits per integer.

BitPacking gem

I've packaged this up into the bitpacking gem.

I hope you find this useful!

Efficient Hash Lookups: Adventures with Benchmarks!

chris — Wed, 16 Oct 2024 00:20:34 GMT

Efficient data retrieval is crucial for high performance applications. Whether you're configuring a complex system or handling large datasets, the speed of hash lookups can significantly impact performance. In this post, we'll explore various methods to optimize hash lookups and benchmark their performance.

You can find my benchmarking code available here.

The Challenge

Let's say that you have a hash (aka dictionary aka map aka associative array; something with key/value pairs) representing configuration for your system.

At runtime, let's assume that your application only needs access to a small number of these configuration items to handle a given request.

Let's also say that this hash can get pretty large; some users have very complex configurations.

{
  "setting_1":     "value 1",
  "setting_2":     "value 2",
  /* ... */
  "setting_10000": "value 10000",
}

A common way of storing this configuration is in a JSON file. JSON is really simple to use for this type of problem. Just use something like JSON.parse to parse your file, and do a lookup for whatever key you want:

config_data = File.read("config.json")
config = JSON.parse(config_data)
puts config["setting_500"]
# => "value 500"

Awesome, what's the problem?

Assuming that we just want to fetch a small number of items from this configuration, there are a few problems here:

We have to load the entire file into memory
We then have to parse the entire file to get our hash object. This involves inserting every item in configuration into a new hash object, which is an O(n) operation.

If our configuration is quite large, then the parsing & hash construction time will be far greater than our lookup time.

$ ruby benchmark.rb -f json -s hash
Calculating -------------------------------------
                load    829.185 (±30.0%) i/s    (1.21 ms/i)
               parse     24.779 (± 8.1%) i/s   (40.36 ms/i)
             get_key     4.242M (± 3.7%) i/s  (235.72 ns/i)
  load/parse/get_key     22.561 (± 8.9%) i/s   (44.32 ms/i)

(I've removed some superfluous information from the output snippets here in the interests of clarity; the number I'm paying attention to is the time per iteration represented in the last column.)

Lookups into the hash, once created, are very fast: about 200ns per lookup.

However, for 100,000 entries, just parsing the data and creating the hash takes about 40ms on my system.

In fact, the hash construction time is the fundamental issue to overcome when trying to speed up our overall time here. Simply creating a hash from the data takes about 20ms on my system:

$ ruby benchmark-to_h.rb < output/array.json
Calculating -------------------------------------
                to_h     37.764 (±18.5%) i/s   (26.48 ms/i)

It makes sense when you think about it. Before we can get our hash object containing our configuration, we need to iterate through each key/value pair and insert it into the hash object. Only then can we do a lookup for the key we're interested in.

How can we make this better?

A bunch of different ideas come to mind:

We can try different ways to encode the data. For example: using MessagePack, protobufs, or capnproto. Each of these use more efficient ways to serialize/de-serialize the data, speeding up the parsing time.

Another idea is to encode our data as a sorted list of key/value pairs instead of as a hash, and use a binary search to lookup the key that we want.

Or, we can try and use perfect hashes; here we would encode our data as a list of key/value pairs as above, but augmented with additional information that makes it possible to treat the array as a pre-computed hash table.

Other ideas: sqlite, dbm/lmdb. These benefit from encoding the index within the file itself, so we don't need to reconstruct a hash table/index again when loading the data. Storing more structured data in these is more challenging though, and lookups in these types of data stores are typically O(log N) instead of O(1)

MessagePack

MessagePack is an easy thing to try first since it's almost a drop-in replacement for JSON in our use case.

$ ruby benchmark.rb -f msgpack -s hash
Calculating -------------------------------------
                load    679.466 (± 5.3%) i/s    (1.47 ms/i)
               parse     27.958 (± 3.6%) i/s   (35.77 ms/i)
             get_key     4.692M (± 3.4%) i/s  (213.13 ns/i)
  load/parse/get_key     27.031 (± 3.7%) i/s   (36.99 ms/i)

Well, we've improved the parsing time by about about 5ms, which is a nice improvement, but still pretty slow overall. Again, we have that large hash construction time to overcome, which MessagePack doesn't help us with.

Protobufs

To experiment with Protobufs, I've created a simple definition using the native map type:

syntax = "proto3";

message PBHash {
  map<string, string> data = 1;
}

The results here are surprising:

$ ruby benchmark.rb -f pb -s hash
Calculating -------------------------------------
                load      1.311k (±35.9%) i/s  (762.93 μs/i)
               parse      21.496 (± 4.7%) i/s   (46.52 ms/i) # !!
             get_key      3.098M (± 3.2%) i/s  (322.75 ns/i)
  load/parse/get_key      20.707 (± 4.8%) i/s   (48.29 ms/i)

It looks like parsing protobuf maps are significantly slower than parsing the equivalent JSON or msgpack objects.

Encode as an array

Let's see what happens if we encode our data as an array.

This may improve parsing time, since we don't need to construct a large hash object in memory. The trade-off is that lookup times become O(log n) instead of O(1), assuming we can sort our input. As a first approximation, we can compare times to load & parse the array.

Using JSON I see marginal improvement to load & parse the data (~30ms), but using Protobufs we can load & parse the data in about 6ms. Unfortunately I couldn't get good measurements for the full load/parse/get_key with protobufs; it seems like doing a binary search for elements in the protobuf arrays is relatively slow in Ruby. Fetching individual items is fast.

ruby -Iproto benchmark.rb -f pb -s array
Calculating -------------------------------------
                load     1.641k (±10.7%) i/s  (609.32 μs/i)
               parse    163.886 (± 3.1%) i/s    (6.10 ms/i)
             get_key     27.325 (±11.0%) i/s   (36.60 ms/i)
  load/parse/get_key      3.000 (±33.3%) i/s  (333.32 ms/i) # ???

Perfect hashes

Maybe we can get the best of both worlds by using a perfect hash to order our data in a particular way to make it cheap to find?

There are several ways of constructing a perfect hash; the CHD algorithm is fairly easy to implement (without the "compress" part of compress-hash-displace) , and can compute a minimal perfect hash function in O(n) time.

The way it works is that we compute a secondary array of seeds that are used to prevent collisions between keys in the data set. To find the index of an element by its key, we compute:

# Find index for `key`
seed = seeds[hashfunc(seed: 0, data: key) % seeds.size]
index = hashfunc(seed: seed, data: key) % data.size]
value = data[index]

The results of using CHD with Protobufs are very good (using cityhash as the hashing function):

$ ruby benchmark.rb -f pb -s chd
Calculating -------------------------------------
                load     1.920k (± 7.0%) i/s  (520.85 μs/i)
               parse    173.076 (± 4.0%) i/s    (5.78 ms/i)
             get_key   491.510k (± 1.9%) i/s    (2.03 μs/i)
  load/parse/get_key    154.828 (± 1.9%) i/s    (6.46 ms/i)

Summary

Wow, that was a lot! What did we learn?

Benchmarks are hard! I wouldn't take any results here as absolute proof that X is better or worse than Y. Your data and use cases are most likely very different.

Using perfect hashes or binary search in an array of key/value pairs look like they're much faster in general than simple lookups into a hash object. The time to construct the hash object plays such a large part in the overall timings.

I'm surprised at the performance of protobufs in generally. They're much slower than I expected, in everything except the perfect hash (CHD) case. It makes me wonder if something is wrong with my implementation, or Ruby's protobuf gem.

$  ruby benchmark-final.rb
Comparison:
load/parse/get_key pb chd:        124.3 i/s
load/parse/get_key msgpack array: 50.2 i/s - 2.48x  slower
load/parse/get_key msgpack chd:   41.1 i/s - 3.02x  slower
load/parse/get_key json array:    37.3 i/s - 3.33x  slower
load/parse/get_key json chd:      27.9 i/s - 4.46x  slower
load/parse/get_key msgpack hash:  23.8 i/s - 5.22x  slower
load/parse/get_key json hash:     20.7 i/s - 6.00x  slower
load/parse/get_key pb hash:       19.4 i/s - 6.40x  slower
load/parse/get_key pb array:      3.1 i/s - 40.07x  slower

To follow up, I'd like to explore using capnproto, and running similar benchmarks in other languages to see if the protobuf performance improves.

Extrakto!

chris — Sat, 02 Mar 2024 01:50:17 GMT

One of my favourite tmux plugins is extrakto.

It's this amazing little plugin that lets you insert text by fuzzy finding within the current tmux session.

Why is this useful?

Think of how often you copy/paste some line of text from the terminal in order to pass it into another command. For example, the output of git status, or perhaps your build or test failed and you want to open that file in your editor (nvim of course!).

Instead of taking your fingers off your keyboard, extrakto lets you quickly find the text you want and add it at the current cursor position.

Using it is simple: just type your prefix (Ctrl-b for me), then tab, type your fuzzy search, and hit tab again to insert the text.

Pull Diagnostic Support for Neovim

chris — Tue, 06 Feb 2024 15:31:10 GMT

I've used vim as my primary text / code editing tool for...literally decades. A few years ago, I tried out neovim, hoping to take advantage of more advanced developer oriented features like treesitter and native LSP support.

At Shopify, I do a lot of work in Ruby. Our incredible developer experience team maintain ruby-lsp, a language server for the Ruby language.

Of course, I wanted to make sure that using it in neovim was a great experience!

LSPs can do many things: auto completion, enabling jumping to function or type definitions, showing function documentation, code formatting, and displaying code linting results.

In this particular instance, I wanted to take advantage of ruby-lsp's "diagnostics" support, which provides real-time feedback about syntax or linting errors (via rubocop)

For example:

brief aside: I find the nomenclature here is a bit awkward. LSP is an abbreviation for "Language Server Protocol". neovim implements a client which implements the LSP, which connects to one or more servers with also implement the protocol. Commonly a server which implements the LSP protocol is referred to as an "LSP server" or "LSP." I'll do that here.

Push vs pull

Unfortunately for me, neovim before 0.10 only supports "push" diagnostics, and ruby-lsp only supports "pull" diagnostics.

"Push" diagnostics are where the LSP pushes diagnostic information about your source code to the client (neovim in this case). The challenge with push diagnostics is that it's difficult for the LSP to know when to generate and send the diagnostics. It has no way to coordinate with other LSPs, and doesn't really how the user is interacting with the editor, aside from receiving events when the sourcecode has been changed. This becomes a problem when many changes to a file are made; how does the LSP know when to generate and push the diagnostics? Should it do it on every change? Wait for a bit after the last change is made?

These challenges are why the language server protocol was updated to add support for "pull" diagnostics. In this model the client (neovim) requests diagnostics from the LSP server, which is much more efficient since the client (neovim) can make much better decisions about when to perform sourcecode diagnostics.

more technically, "push" diagnostics are implemented via the textDocument/publishDiagnostics method. "pull" diagnostics are implemented via the textDocument/diagnostic method

Pull support in neovim (open-source FTW!)

Last year I spent some time adding support for pull diagnostics to neovim 0.10. I learned a lot about neovim, LSP, and lua in the process!

To me, this really highlights the benefits of open-source: there was something in my toolbox that I wasn't happy with, and I am empowered (and encouraged!) to contribute and make it better for everyone! Of course, I relied heavily on more experienced contributors to the project, and, of course, my initial implementation required some follow-ups.

My first neovim plugin

However, neovim 0.10 hasn't been released yet, and so I also wanted to make sure that I could use ruby-lsp with stable (0.9.x) versions of neovim.

So I created my very first neovim plugin: https://github.com/catlee/pull_diags.nvim. This enables "pull" diagnostics for neovim prior to 0.10. It's not quite as efficient as the support built into 0.10, but it works for my day to day development. If it doesn't work for you, please let me know!

Installing it is quite simple; for example, with Lazy.nvim:

{
  "catlee/pull_diags.nvim",
  event = "LspAttach",
  opts = {},
}

Final thoughts

As a professional software engineer, I think it's important to invest in the tools that I use to get my work done. Concretely, this means that I need (and want!) to spend time learning what new tools and techniques are out there now, and to make sure to sharpen my tools.

In this case, I wanted to take advantage of a new technology (LSP & ruby-lsp) in my editor of choice (neovim). Since these tools are all open-source, I'm empowered to change them to fit my needs.

In other words, as professionals, we shouldn't accept drawbacks or limitations in our tools: we should fix them! It's just another piece of software after all!