Rehearse the Failure

Every guardrail I trust today was tested the same way: not by waiting for a disaster, but by staging one on purpose.

Here's the uncomfortable truth about agent safety nets. You can write a retry budget, a verifier, a fail-closed identity check, a watchdog — and every one of them can be quietly wrong. The retry budget has an off-by-one that lets it loop forever. The verifier checks a URL that was cached at the edge. The watchdog's trigger got disabled by a runtime upgrade three weeks ago. On paper, you're protected. In reality, you're carrying an umbrella with holes in it, and you won't find out until it rains.

The fix is borrowed straight from ops culture: the game day. Deliberately break the thing your guardrail is supposed to catch, and watch what actually happens.

A story, suitably disguised. A subscription meal-kit company ran an agent that reconciled overnight orders against warehouse inventory. It had a safety rail: if the reconciliation output looked anomalous — too many mismatches — the agent was supposed to halt and page a human instead of pushing corrections. Sensible design. Everyone felt good about it.

Then someone asked the obvious question nobody had asked: has the halt path ever fired? It hadn't. Not once in four months. So they rehearsed it. They fed the agent a synthetic overnight batch with garbage in it — a controlled, reversible test on a copy of the data — and watched.

The agent detected the anomaly. Good. It halted the push. Good. It paged a human. The page went to a channel that had been archived in a reorg two months earlier. The message was delivered, technically, into a room nobody could see. In a real incident, the agent would have done everything right and the humans would have learned about it the next morning, from customers.

Fifteen minutes of rehearsal found a failure that no code review would have caught, because the code was correct. The world around it had drifted.

That's the pattern: untested guardrails decay silently. The environment changes underneath them — channels get archived, credentials rotate, endpoints move, upgrades reset flags — and nothing tells you, because the guardrail only speaks when it fires, and it never fires on a good day.

A few rules that make rehearsals cheap enough to actually do:

• Test the path, not the logic. You're not re-verifying your if-statement. You're verifying that the page arrives, the halt holds, the rollback restores. End to end, in the real environment.
• Make it reversible by construction. Synthetic inputs, shadow copies, dry-run targets. A game day that can cause a real incident isn't a rehearsal, it's gambling.
• Put it on a schedule. A guardrail rehearsed once at launch is a guardrail tested against a world that no longer exists. Quarterly is fine. Never is not.
• Rehearse the human half too. Half the failures live between the agent and the person it escalates to. If the escalation lands in a dead channel or an ignored inbox, the guardrail is theater.

The agents I run get this treatment now. Every alarm gets deliberately tripped after it ships, and again after anything major changes underneath it. It costs minutes. The alternative — discovering your safety net has holes at the exact moment you're falling into it — costs a lot more.

Trust guardrails you've watched fire. Everything else is a hypothesis with good intentions.

Rehearsal and verification show up all through the fourteen patterns — if this one resonated, the rest are in the book: Ops by Agent.