The outage that moved us to Cloudflare

Postgres wrote its last log line at 06:02:22. Three minutes later every Cloud SQL metric for TipBop’s production database flatlined, and every connection from the app timed out. Nobody had deployed anything. The serving revision was two days old. Google’s status page said everything was fine.

It was not just TipBop. Two TyreStack databases in a different project, same zone, went dark in the same minute. That was the first useful clue: this was europe-west2-b having a bad morning, not anything we had done.

Why it was in one zone at all

Both products were in beta. TipBop had no external customer, and TyreStack had no signed fitter. In that state we had made a deliberate call, written into TipBop’s decision log on 1 October: run the database in a single zone, on a shared-core machine, and do not pay for high availability yet. The log gave the reasons. There was no external customer to hurt. Deletion protection was on. We had drilled a restore and a point-in-time recovery clone the day before, and the clone took 32 minutes and matched production. And high availability roughly doubles the database line on the bill. We set ourselves a trigger: review by 31 December, or before the first external customer, whichever came first.

The obvious objection is that a payments app on one zone of a shared-core database is a bad look, and it would be if it had customers. Google’s own advice is not to run production on a shared-core tier, and it carries no uptime promise. We knew that. The trade was that, in beta, a day of downtime would cost us our own time with no external customer waiting, while a permanent high-availability bill bought protection for customers who did not exist yet. The outage arrived before either of our triggers did.

The same morning’s review wrote the rule down for later: a production database that holds real customer data or money is regional, on a dedicated core, from its first real customer. Shared-core and single-zone are for staging and beta only. So the argument is not that one zone is fine. It is that one zone was right for that stage, and that the stage had an exit condition written down in advance.

The morning, hour by hour

On 20 October the gap came due.

The restart at 06:42 took twenty minutes and was a false dawn. Postgres came back but nothing could reach it, and write-ahead logging was still failing. Rather than wait an unknown number of hours for Google, we restored the 04:29 backup into a fresh instance in zone a.

That decision had a cost we could measure. Anything written between 04:29 and 06:02 was at risk. The request logs for that window showed no writes at all, only bots poking at paths that do not exist, and a full diff later proved it: every primary key and a checksum across every table, zero rows missing.

The restore finished at 07:21. We did not cut over until 09:06. The cutover script sat in a wait loop because a filter in one gcloud command matched nothing, so it waited politely for a condition that could never be true. One hour and forty-five minutes of downtime was ours, not Google’s. That script now fails loudly instead of waiting forever.

The review said no

While TipBop limped back, we asked a harder question: should we be on Google Cloud at all? We wrote an adversarial review that morning, deliberately trying to talk ourselves out of a knee-jerk move. It was good, and it said no.

Its argument went like this. The outage hit a zonal, shared-core database. Nothing about it says “change cloud”. It says “pay for high availability on the one product that takes money”, a bill of about £45 to £60 a month instead of about £12. It pointed out that Cloudflare had several outages in 2025 that would have hurt us, the longest lasting hours, and that every one of our products already sits behind Cloudflare’s DNS and proxy. Our real single point of failure was Cloudflare already.

That is a fair argument and we still think it is mostly right as written. We overruled it anyway, for three reasons.

First, cost at our size. TipBop’s entire Google bill the month before was £12.61. The fix the review proposed would have multiplied that by four before a single extra customer arrived. TyreStack’s projected bill at twenty customers was £150 to £250 a month on Google with high availability, against £5 to £15 on Cloudflare.

A fair challenge is that a day of work to save about £10 a month on the bill we actually had, or more than £40 against high availability, is a bad trade. If money were the only reason, it would be. It was not the main one. The effort of the port was not our constraint, so the question was which end state gave us the lowest bill, the fewest things to fail and the quickest restore. The bill was the tiebreaker, not the case.

Second, fewer moving parts. On Google we ran a container platform, a managed Postgres, a connection proxy, a storage bucket and a Cloudflare Worker in front of all of it. Each one is a thing to patch, monitor and pay for. On Cloudflare it is one Worker and one database, behind the proxy we already trusted with everything.

Third, and this is the one people underrate, restore speed beats uptime promises. The question that actually hurt that morning was not “how often does it break?” but “how fast can we put it back?”. We wanted a setup where putting it back was a script we had already run, not a support ticket.

Why it was not reckless

Moving a payment app in beta between clouds in an afternoon sounds like the kind of thing that ends up in a postmortem. Three things made it boring instead.

The old system stayed up and untouched until the new one proved itself. The production Worker was deployed “parked”, answering only a canary hostname, with its payment sweepers switched off so two systems could never pay out at once.

We replayed real traffic before trusting it. Twenty-four hours of real requests, 1,282 distinct paths, were sent to the new Worker running on a copy of production data. 1,273 came back byte-identical. The other nine differed by design, and each was written down with its reason.

And we rehearsed the undo before doing the thing. Rolling back meant reloading every table into a scratch Postgres and swapping the hostnames back. We timed it.

The first real cutover attempt, at 13:52, rolled itself back. Five checks for alias domains like www.tipbop.com failed because they were being tested while the site was deliberately in maintenance mode. The script did exactly what it was built to do: it put the hosts back on the old system in about three seconds and stopped. We fixed the check, merged it, and ran it again. Attempt two finished at 14:42, all ten hostnames moved, 230 seconds end to end, every checksum matching.

An hour later we deleted the old Cloud Run service, the Cloud SQL instance, the container registry and the staging project. Leaving them running for the planned two weeks would have been the cautious move. But nobody was using the old system, keeping it meant paying for it and patching it, and we had a nightly off-platform backup, 30 days of point-in-time recovery on D1, and a final Postgres dump in cold storage. Caution has a running cost too.

What we changed for everyone

TipBop was first. TyreStack followed the same day, and ZooLog had been on Cloudflare since its first commit. That evening it became a house rule: every new service, database and background job goes on Cloudflare, with EU data residency set at creation on anything holding personal data. Google Cloud is now an exception we have to write down and justify, not a default.

We kept one thing from the review that argued against us. Cloudflare is now a bigger single point of failure than it was, and we treat it that way. Every production database has a nightly export that lands outside Cloudflare, and a restore of that export is tested automatically every night. If Cloudflare has a six-hour day, we lose six hours. We do not lose the data.