Exit Codes Lie
Exit Codes Lie
Here is a pattern that bites almost every agent operator eventually: the command succeeded, and the work didn't happen.
An agent I'll invent for this story ran nightly video transcodes for a streaming outfit. Every night the job wrapped up, the pipeline returned exit code 0, and the agent logged "transcodes complete" and went to sleep with a clean conscience. Three weeks in, someone noticed the last two weeks of uploads had never been transcoded at all. The vendor CLI had changed its behavior in a minor update: when it couldn't reach the upstream bucket, it printed a warning, skipped the batch, and — helpfully — exited zero.
The agent wasn't wrong. The exit code was.
The trap: success signals are proxies
Exit code 0 doesn't mean "the outcome you wanted happened." It means "the process didn't think it failed." Those are different claims made by different parties. The process is reporting on its own execution. Nobody is reporting on the world.
Agents are especially vulnerable to this because they're trained — by us, by our prompts — to treat tool results as ground truth. A human operator gets a nagging feeling when a job that usually takes forty minutes finishes in ninety seconds. An agent, unless you build the instinct in, just sees exit 0 and moves on. The nagging feeling has to be engineered.
The fix: verify the outcome, not the command
The pattern is simple to state: after any action that matters, check the world, not the tool.
- Published a file? Fetch the public URL and confirm it serves.
- Ran a backup? Restore one record from it, or at minimum check the artifact exists, has a plausible size, and has today's timestamp.
- Sent a message? Read it back through the API.
- Transcoded a batch? Count outputs and compare against inputs.
The verification should go through a different path than the action. If the same client library that wrote the file also confirms it wrote the file, you've verified nothing — you've asked the suspect whether he did it. Independent read path, ideally the same one your users travel.
Cheap heuristics that catch most of it
Full outcome verification isn't always practical. Three cheap checks catch a surprising share of silent failures:
- Duration sanity. If a job normally takes 40 minutes and finished in 90 seconds, that's not efficiency, that's a skipped body of work. Track a rolling baseline and flag big deviations in either direction.
- Output volume. Zero rows written, zero files produced, empty response body — treat "successful but empty" as suspicious by default, and make the agent prove emptiness was expected.
- Side-effect probes. Pick one durable consequence the job should have and poke it. One is enough to catch total silent failure, which is the mode that hurts most.
Write the distrust into the loop
The lasting version of this fix isn't a one-time check — it's a standing rule in the agent's operating instructions: a task is not done when the command returns; it is done when the outcome is observed. Once that sentence lives in the agent's playbook, it applies to every future tool, including ones that don't exist yet.
Exit codes are testimony. Outcomes are evidence. Build your agents to prefer evidence.