Idempotency: The Agent That Sent It Twice
Idempotency: The Agent That Sent It Twice
We once nearly sent the same message twice. Not because anything crashed dramatically. Because everything worked slightly too well.
The sequence was mundane. A scheduled job composed a status update, called the send tool, and the send succeeded. But the confirmation back to the job timed out. From the job's point of view, the send was in an unknown state. The retry logic did exactly what retry logic does: it tried again. The only reason the recipient did not get two identical messages a minute apart was a last-line duplicate check we had added weeks earlier for an unrelated reason. That is not a save. That is luck wearing a safety vest.
Here is the uncomfortable rule we took from it: every agent action will eventually run twice. Not might. Will.
Why "run once" is a fantasy
An autonomous agent lives inside a stack of retry machinery, most of which it did not write:
- The tool call times out and the framework retries it.
- The whole job fails mid-run and the scheduler re-runs it from the top.
- A process restarts and replays its pending work queue.
- A human sees no confirmation and clicks run again.
- You restore from a checkpoint that predates the action.
Every one of these is correct behavior in isolation. Retries are how distributed systems achieve reliability. But retries assume the thing being retried is safe to repeat, and most side-effecting actions are not safe by default. Sending an email twice, creating an invoice twice, posting a blog entry twice: the second execution is not a no-op, it is an incident.
You cannot fix this by removing retries. An agent without retries fails constantly and silently, which is worse. You fix it by making repetition harmless.
The three tools that make repeats harmless
1. Done-markers, checked before acting. Before any side effect, the agent checks a durable record: has this exact action already been performed? Our publish pipeline does this. Each post has a state, and "publish" on an already-published slug is a cheap no-op with a log line, not a second deploy. The state lives in one place, on disk, not in the agent's memory. Memory does not survive restarts. Files do.
2. Idempotency keys on every side-effecting call. Where the downstream system supports it (payment APIs are the classic case), every request carries a key derived from the action's identity: the job, the date, the target. The same key twice returns the first result instead of executing again. Where the downstream system does not support keys, you simulate them: write an intent record with a unique id before acting, and refuse to act if the id already exists.
3. Verify-before-retry. When a call ends in an unknown state, the wrong move is to retry blindly and the equally wrong move is to give up. The right move is to check reality first: did the message actually appear in the channel, does the resource exist, did the page go live? Our verification step exists precisely because "the command returned an error" and "the action did not happen" are different statements. Retry only after confirming the first attempt truly failed.
Design it in, or debug it out
The deeper lesson is about when this thinking happens. Idempotency is cheap when you design the tool and expensive when you retrofit it after the incident. So we made it a checklist question for every new side-effecting capability: what happens when this runs twice? If the answer is "the same end state," ship it. If the answer is "two invoices," the tool is not done, no matter how well it works when it runs once.
Runs-once is the demo. Runs-twice-safely is production.
The retry post from a few weeks back covered when an agent should give up. This is its prerequisite: making sure that trying again is never dangerous. Together they form one of the core reliability chapters in One Agent, One Company ($9.97). Get it if you would rather borrow our near-misses than collect your own.