Engineering
How we build and run things: architecture, infrastructure, tooling and the trade-offs behind them.
2026
What Cloudflare can still do to us
Decision recordWe moved TipBop and TyreStack to Cloudflare in a day and made it the default for everything. This is the case against that, from our own standard: five ways it fails, the limits we accepted, and four conditions that would send us back.
Every guard on our agents started as an incident
Deep diveOur AI coding agents run behind five small scripts that can refuse a shell command before it runs. None of them was designed up front. Each one is the scar from something that went wrong, and each says so in its first lines.
Load average 14, and the CPU was mostly idle
PostmortemOne of our Macs crawled, with load readings up to 30 and eight agent sessions open. We blamed the cores. Measured at a load of 8 to 14, they were 68 to 80% idle. The memory had run out, and our first fix watched the wrong number.
Who watches the alerts?
Deep diveOn 18 October we found our Slack bot tokens had died and every alert since had gone nowhere. We rebuilt the path so it is much harder to fail quietly, then spent a week learning that an alert path can also fail loudly, by paging for things nobody could fix.
Postgres to D1, with the rollback timed first
Deep diveD1 has no interactive transactions, and TipBop's payouts relied on them. How we moved a tipping app in beta off Postgres in a day: a lease row for the locks, one schema for 37 migrations, and a rollback we timed before we needed it.
Ten thousand worlds before the first real fitter
Deep diveTyreStack's scheduler had no customers yet, so before launch we built an audited oracle and threw ten thousand generated worlds of bookings, holds and late vans at it, comparing the old engine and the new one call for call.
The outage that moved us to Cloudflare
Decision recordAt 06:02 a Google Cloud zone stopped answering and took two products, both still in beta, with it. Our own review said stay and pay for high availability. By 15:45 we had left anyway. Here is the reasoning, and the objections to it.
24 files we are not allowed to edit
Deep diveEd proved his scheduling engine against ten thousand simulated worlds. TyreStack imports it unchanged, locks 24 files to his exact bytes with a hash, and the gate fails the moment one of them differs.
Sign-in without passwords, SMS or an auth vendor
Decision recordOn 16 October we wrote one sign-in standard for every app we run: an emailed code plus Apple, Google and Microsoft, our own revocable sessions, and no passwords, SMS codes or auth vendors. Here is why, and the linking rule that matters most.
A thousand synthetic finger drags
Deep diveA bot playing our iOS puzzle game through real touch events caught SwiftUI dropping finger positions whenever a frame ran late, turning a +672 chain into +80. It also caught a Release build that no longer compiled.
A green health check and an empty database
PostmortemTipBop moved to Cloud Run on 17 September with a test-mode Stripe key, placeholder secrets, no data and no photos. The health check said ok the whole time. What it was checking, what it should have checked, and the two guards we added.