Reliability Is Not a Permission Slip
Reliability Is Not a Permission Slip
There is a moment that arrives with every agent that works well. It has run some capability hundreds of times without a mistake. The log is spotless. And someone asks the obvious question: it has earned this, why is a human still clicking approve?
The question feels fair. It is the same logic we apply to people, where a track record is exactly how trust gets built. But applied to the boundary between reversible and irreversible actions, it is the most expensive reasoning error available, because it sounds like good engineering judgment right up until the incident.
Here is the thing we had to get explicit about: reliability on reversible work and authority over irreversible work are different currencies, and one does not convert into the other.
What the track record actually proves
An agent that has edited four hundred drafts without incident has demonstrated something real. It has proved that its edits are usually good, and more importantly, that the correction loop works — bad edits got noticed and fixed cheaply, four hundred times.
Look closely at what carried that success. Every one of those four hundred actions had an undo. The reason the failures were invisible is not that they never happened. It is that they were absorbed. A wrong draft edit gets rewritten before anyone reads it. The safety net did its job so quietly that the record now looks like pure competence.
So the spotless log is evidence about two things fused together: the agent's judgment, and the cheapness of being wrong. When you move that agent to an irreversible action, you keep the first and throw away the second. The track record was never measuring the thing you are now relying on.
Why the base rate is the wrong comparison
The seductive version of the argument is statistical. If the error rate is one in four hundred, and a human reviewer misses things at some rate too, is the agent not the safer reviewer?
That comparison quietly assumes every error costs the same. It does not. On reversible work, an error costs a retry. On irreversible work, the same error costs a retracted claim to a customer, a payment that has to be clawed back, an email that has already been read and cannot be unread. You cannot average across those. A one-in-four-hundred chance of rewriting a paragraph and a one-in-four-hundred chance of sending the wrong number to a client are not the same risk wearing different hats.
Low probability times unbounded, unrecoverable cost is not a small number. It is a number you have declined to compute.
The failure mode is silent accumulation
The real damage is not one bad decision to remove one gate. It is the drift.
When we went back and audited our own capability list, we found two tools sitting in the irreversible tier with no gate in front of them. Nobody had removed a gate. No decision was ever made. They had simply been built during a stretch when everything was working, they had never been wrong yet, and their clean history read as permission. Absence of an incident had been silently promoted to evidence of safety.
That is the mechanism to watch for. Gates do not usually get argued away. They get skipped during a good week and never revisited, and the longer the good week runs, the more reasonable the omission looks.
What we do instead
Three rules, and the third is the one that actually holds the line.
Permissions attach to the action, not the performer. The gate is a property of what could happen, decided when the capability is designed. It does not have a field for how well the agent has been doing lately.
Track records buy speed, not scope. A strong history is a real reward, and it should be paid out in the currency it earned: less logging overhead, fewer verification steps, faster paths on reversible work. It never buys a new tier.
The gate is reviewed on its own schedule, never in the glow of a success. If an irreversible action should become autonomous, the honest way is to change the action — build the undo, add the soft-delete window, convert the send into a draft, put a hold in place of a charge. Make the thing reversible, and it moves tiers legitimately. Asking for the promotion because the agent has been good is asking to be paid in the wrong currency.
The compressed version
A clean record on cheap mistakes tells you the recovery worked. It tells you nothing about a world with no recovery. If you want an agent to have more authority, do not hand it a permission slip for good behavior. Go build it an undo, and let it earn the tier it can actually be held to.
Deciding what an agent may do alone is the whole job of running one. The tier model, the gates, and the audit we used to find our own unguarded tools are in One Agent, One Company ($9.97).