For about a day from 17 September, TipBop’s production server answered /readyz with ok. The database was connected. Stripe accepted the key. Every box on the check was ticked.
The Stripe key was a test key. The page that takes a tip had an empty publishable key, so Stripe’s card form could not even load. The webhook and email secrets were placeholders. The database had the right schema and no rows. The photo bucket was empty. Web push was off, and most of the environment variables the old server had set were simply absent. Nobody could have tipped anybody, and nothing told us.
How it got that way
TipBop had been running on a single server of ours. On 17 September we moved it to Google Cloud Run, which meant writing down, as a deploy script, everything the old server had provided by being the old server.
That script did not write all of it down. A setup script we had committed seeded a Stripe test key and placeholder values so a fresh environment would boot. Production booted on them. The data stayed behind on the old machine, and the photo bucket on the new side was created empty.
None of this was hidden. Each fault was one command away from being seen. We just had no command that looked.
The fair reaction is that shipping to production on a test key and an empty database is a beginner’s mistake. The mistake is real, and the cause is ours: a setup script built for fresh test environments was allowed to seed production. What contained it was the stage. TipBop was in beta, so no one outside the company was waiting on a tip to go through. The cost was a day of an app that could not take money, not a day of someone’s takings. The same fault with customers would have been an incident. That is why the fixes below are not “be more careful”, but code that refuses to start in the wrong world.
The readiness check was not broken. It did what readiness checks do: proved the process was up, could reach its database and could talk to Stripe. A test-mode key talks to Stripe very happily. The check answered “is it running?” and we read the answer as “is it right?”.
The restore
The database came back from the old server’s nightly backup, taken at 03:11 UTC on 17 September. The old database had been dropped at 14:09 that afternoon, so that archive was the copy that mattered. Tips taken between 03:11 and the cutover were not in it. They did not need to be: Stripe is the ledger of record, and TipBop’s reconcile sweeper rebuilds tips from Stripe’s events. The Stripe dashboard was the cross-check for that window.
The real secrets went into Secret Manager by pipe, never printed, and each was checked by prefix only. The webhook endpoints are keyed by URL and the URL had not changed, so the old signing secrets were the right ones. By the evening of 19 September the service was on a new revision and the verify script below was green.
Two guards, so it cannot happen quietly again
The server refuses the wrong world. Production already refused to start with a live Stripe key unless a flag said so on purpose. We added the other direction. In production, a test key is now a start-up error unless a second, explicitly named flag is set for a deliberate rehearsal. There is a test for it. This is the shape:
// In production, a Stripe key from either world needs a deliberate flag.
if (isProduction) {
if (secretKey.startsWith('sk_live_') && !flag(env.ALLOW_LIVE_KEYS)) {
problems.push('refusing a live Stripe key: set ALLOW_LIVE_KEYS=1 deliberately');
}
if (secretKey.startsWith('sk_test_') && !flag(env.ALLOW_TEST_KEYS)) {
problems.push('refusing a test Stripe key in production: it takes no real money');
}
}
Only the prefix says which world a key charges in, so the prefix is what the guard reads. The setup script now seeds REPLACE_ME instead of a test key, which the guards also refuse.
Every deploy reads the live service back. The deploy script now declares all 31 variables and 11 secrets the old server set, and a test treats that list as the contract. On its first run it caught two faults we had not spotted: a cookie secret that nothing reads, and the email token fed under a name the mailer never looks for. Email would have stayed dead with a real token in place.
After every deploy, a verify script reads the live service and fails the deploy unless the publishable key is live-mode, the web push key is present, readiness reports the database, Stripe and the reconcile sweeper, the sign-in providers are all offered, every declared variable is on the live revision, and the secret key’s first eight characters say sk_live_. The key itself is never printed. It ends on one of two lines: “verified: this is production” or “NOT production”.
And the error reporting nobody missed
The same week showed us a quieter version of the same problem. Until 13 September the server reported no exceptions anywhere. Nobody noticed, because the symptom of no error reporting is silence, and silence looks exactly like no errors.
On 13 September we wired up Sentry, and chose not to install its SDK. The payment server has four runtime dependencies, and the SDK brings OpenTelemetry and dozens of other packages into the process that handles card payments, to feed tracing and profiling features we had switched off. We wrote a small client that posts the one thing we wanted instead.
On 18 September we dropped Sentry altogether. Cloud Run already reads structured logs from standard output and groups errors from the stack traces in them. So the server writes one JSON line per error, through the same redactor that strips card numbers from every other log line, and on an uncaught exception it logs and exits with no network call on the way out. That is zero new dependencies and no second vendor holding our errors.
Then we made it prove itself. On 19 September we fire-tested the alert paths end to end, the error alert and the failed-backup alert, and only counted each one once its email had arrived. Reading the configuration is not proof. A message in the inbox is.
Update, 7 November 2026
The lesson became rule 3 of the rules every project we run follows: a health check proves reachability, not correctness, and verification means the repo’s own verify script. TipBop moved again, from Cloud Run to Cloudflare, after a Google Cloud zone outage. When we wrote up our hosting risks in early October, TipBop was still in beta with no external customer.
