system-design

I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year.

I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.

They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.

Jedberg on avoiding daylight savings time at reddit.

A good instroduction to observability that covers metrics, traces and logs with examples from none other than the OpenTelemetry documentation.

Marc Brooker writes about the benefits of versioning over coordination (i.e. locks) in database systems, particularly in distributed ones by using a physical clock.

Adrian Cockroft writes about AB testing and building an "abtest" where "Each customer should be in a small number of tests." and the "abtest service should be used across your entire system, for all experiments that touch a customer."

There are three ways a user is enrolled into a test, (1) new users on sign up, (2) feature-specific test when meeting a condition, (3) and a random selection of existing users.

Discovered via Matt Weagle on Mastodon.

Sharing Adam Leventhal's drawing of what happens if that one dependency from that XKCD fails.

A look into a risk assessment done by the Hachyderm Infrastructure team as a precaution to limit exposure to entities that may be governed by US law. The post breaks down each site and components giving an indication of the impact to service if the component were disrupted, and effort to migrate. There's a lot of interesting detail on what it takes to host a large distributed social network like Mastodon well.

An overview of the design choices behind creating the JavaScript Registry (JSR) which is compatible with yarn, npm and others. One thing that stood out to be was that the website is rendered server-side, the post goes through their design to optimize for Time-To-First-Byte between their rendering and API server backed by a Postgres database.

The rest of the post is also a good read.

How fly.io manages WireGuard peers and loads them just-in-time into the kernel speeding up kernel operations.

Dan Luu writes about using simple architectures to build apps using Wave as an example. The post covers the initial technology choices made to be maintained by a small tem and the tradeoffs in the long run, along with the flexibility it provided to adapt to local markets.

Our architecture is so simple I’m not even going to bother with an architectural diagram. Instead, I’ll discuss a few boring things we do that help us keep things boring.