A Watchdog Must Report Its Own Death
Every serious agentic operation eventually adds a watchdog: a small, dumb process whose only job is to notice when the smart process stalls. Publish pipeline hung? Watchdog barks. Backup didn't run? Watchdog barks. It's the classic safety layer, and it works — right up until the watchdog itself dies quietly.
Here's the failure shape, and it's more common than anyone admits. A runtime gets upgraded. A scheduler config changes format. A permission is tightened. The main pipeline keeps working, because it's the thing everyone tests. The watchdog, which nobody tests, stops firing — and stops firing silently, because a watchdog that never barks looks exactly like a watchdog guarding a healthy system.
At a media operation I'll keep vague, an autonomous publishing pipeline half-shipped a release: the index listed it, the asset behind it returned a 404. The watchdog built precisely to catch stalled publishes had been dead for five days. An upgrade had disabled its trigger. Nobody noticed, because the absence of alarms is what success looks like. The system had a safety net with a hole in it, and the hole was invisible from every dashboard.
The fix isn't a better watchdog. It's a different contract: a watchdog must report its own death.
Concretely, that means designing for positive confirmation rather than silence-as-health:
- Heartbeat outward, not just watch inward. The watchdog emits a periodic "I ran, I checked, all clear" signal somewhere durable. A missing heartbeat is itself an alarm condition — checked by something outside the watchdog's own runtime.
- Dead-man's switch over alerting. Don't ask "did anything fail?" Ask "did the checker check?" If the answer is silence, escalate. Silence is never evidence of health; it's only evidence of silence.
- Test the alarm path, not just the happy path. Periodically inject a synthetic failure and confirm the bark actually reaches a human. An alarm that has never fired in anger is an alarm you know nothing about.
- Put the watchdog's liveness in the runbook's first line. When something half-ships, the first diagnostic question should be "was the checker alive?" — because if it wasn't, every other green light is suspect.
There's a deeper design principle here for anyone building autonomous systems: every layer you add to catch failures becomes a new thing that can fail, and it fails in the most dangerous way possible — by removing the signal you were counting on without telling you. Safety layers don't just need to work. They need to be observably working, continuously, in a way that a five-day gap can't hide.
The operation in question now runs a checker-of-the-checker: a tiny, boring heartbeat that would have turned five silent days into one loud hour. It has fired exactly once since. That one bark paid for the whole design.
If you're building agents that act while you sleep, the question isn't whether you have a safety net. It's whether you'd know the moment the net tore. Verification layers and their failure modes get a full treatment in the book: Ops by Agent.