Who watches the alerts?

On 18 October we asked Slack whether our bot tokens still worked. Every one answered account_inactive. Our dead-man monitor’s own log showed the consequence: every DOWN alert it had tried to post had failed, and nothing had reached the alerts channel. The monitor had been doing its job. The last metre had not.

That afternoon we did the simple thing. The monitor fell back to email, Slack was retired as an alert channel, and every alert went to one shared inbox. Two days later we reversed half of that decision, for a reason worth spelling out: email alone gave us no priority, no way to tag a person, and no thread per incident. Slack is where we actually are. It just could not be the only place.

One relay, two paths

The rebuild on 20 October has one rule at its centre: every automated alert goes to Slack and to email, always both, as two paths that do not wait for or hide each other. Everything that raises an alert, from the dead-man monitor to CI to app monitors, posts to one relay. The relay does the rest.

Every alert must carry a priority, and the sender chooses it. P1 means production is down or wrong now: customers affected, data lost, a security incident, money wrong. P2 is degraded or at risk and needs action today. P3 needs a look this week and pings nobody. P4 is for the record. The relay de-duplicates by a key, so a fault that fires every five minutes is one thread with a counter, not two hundred messages, and a resolve lands in the same thread with the evidence attached.

If the Slack post fails, the email copy says so, marks itself as the primary copy, and the post is retried hourly. If the email fails, Slack says so. Neither path can silently take the other down with it.

Watching the watchers

An alert system that fails silently is worse than none, because you stop looking. So the path is checked in layers, each by something that does not depend on the layer it is checking.

• Every hour, the relay checks that its Slack token is valid, the channel exists, it is not archived and the bot is still a member, without posting anything. Two failures in a row and it emails. Slack cannot report its own death, so it does not have to.
• Every Monday, a real canary alert goes through both paths, end to end.
• Every five minutes, the dead-man monitor, a separate Cloudflare Worker, reads the relay’s status: both paths healthy, a canary within the last eight days. It reports by direct email, never through the relay it is checking.
• Every hour, a watchdog running on GitHub’s own runners checks that the dead-man monitor is still ticking. It does not depend on Cloudflare or on any of our machines. The dead-man had vanished once, which is why this layer exists.

The dead-man monitor does more than ping pages. Many of its targets are silence checks, which fire when something that should happen stops happening: TipBop has received no Stripe webhooks for too long, a dead-letter queue is filling, transactional email has gone quiet, the daily estate checks have not reported in 36 hours. A page that answers 200 can still be broken. A webhook that has not arrived in a day is harder to argue with.

Then the phones

With the path trustworthy, we wanted the important alerts to ring a phone. On 23 October P1 got its own channel and a push notification through ntfy. It failed on the first test. ntfy’s free service limits anonymous publishing per IP address, and Cloudflare Workers share their outbound addresses with everyone else’s Workers, so our first publish hit someone else’s quota and came back 429. ntfy itself cannot run on Workers, and publishing from them needed a paid plan.

The same day we switched to PagerDuty’s free plan: our own account in the EU region, push then phone call, and the alert key reused as the incident key, so a repeat or an escalation never opens a second incident. We also stopped repeats from shouting. A P2 that is still firing now re-posts every four hours instead of every hour, a P3 daily instead of every six hours.

Nineteen P1 alerts, three unplanned, no emergencies

Six days later we counted. Nineteen P1 alerts had been logged since 20 October. Three were unplanned. Two were ZooLog release pipelines failing, which meant a release did not ship while production carried on untouched. The third was a client’s site health check returning 503 because Cloudflare had an open D1 incident: the site itself stayed up, its form database errored, and nothing we could do would have fixed it.

None met the definition we had written ten days earlier. So the severities were wrong, and severity is the switch, so we fixed the severities: a failed release tag became P2, and that client health check became P2.

It took one more false page that afternoon, when the dead-man re-escalated the same check from a target list that had not yet been updated, for us to make the real change. Paging is now default-deny. A P1 always reaches the critical channel and the inbox, but it only rings a phone if its key is on a short allowlist (TipBop readiness, ZooLog’s app health, TipBop’s Stripe webhooks going silent, the dead-man watchdog, emergencies from the client portal) or a director raised it by hand. CI never pages. Third-party outages we cannot act on never page. Adding a key to the list means writing down the customer harm that justifies waking someone up.