The Retry That Made It Worse
The Retry That Made It Worse
Retry is the cheapest reliability feature there is. One config line, and transient failures stop paging anyone. It is also the easiest way to turn a single failure into a compounding one, because retry quietly assumes two things that are not always true.
The first assumption is that the failure was transient. The second, and more dangerous one, is that the failed action was never applied.
The timeout is the problem case
Most failures are honest. A connection refused means nothing happened. A 400 means the request was rejected before it did anything. Retrying either of those is safe, because the first attempt left no trace.
A timeout is different. A timeout says the answer did not arrive. It says nothing at all about whether the work was done. The request may have been dropped on the way out, or it may have been received, processed, committed, and the acknowledgement lost on the way back. From the caller's side those two outcomes look identical.
Retry treats them identically too. It sends the request again. If the work had already landed, it now lands twice.
This is why timeouts produce the strangest incidents: the duplicate charge, the double notification, the record created twice with two different identifiers. Nothing errored. The system did exactly what it was configured to do.
Retry and identity are one feature
The fix is not fewer retries. It is giving the receiver a way to recognize that it has seen this exact request before.
That is what an idempotency key does. The caller generates a stable identifier for the work, not for the attempt, and sends the same one on every retry. The receiver records which keys it has already completed. A repeat arrives, gets matched against the record, and returns the original result instead of doing the work again.
The important detail is that the key must be stable across attempts and unique across distinct work. A key regenerated on each retry is not a key, it is a timestamp. A key reused for genuinely different work silently drops the second one.
Stated plainly: a retry policy without an identity key is a duplication policy. It just has not been tested against a timeout yet.
Which operations are safe to repeat blind
Not everything needs a key. It is worth sorting operations before adding machinery.
Reads are safe. Repeating a read produces the same answer and changes nothing.
Operations that set an absolute value are usually safe. Setting a status to complete, writing a record to a known state, or assigning a specific owner can run twice with the same result.
Operations that apply a relative change are not safe. Incrementing a counter, appending to a list, sending a message, transferring an amount, or creating a new record with a generated identifier all produce different outcomes when they run twice.
The test is simple: if running it twice gives a different end state than running it once, it needs an identity key before it gets a retry.
Bound the attempts and watch the duplicates
Two smaller habits keep retry from compounding in other ways.
Cap the attempts and space them out. Immediate unbounded retry against a struggling service adds load at the exact moment it can least absorb it, turning a slow dependency into an unavailable one.
Then measure for duplicates directly, rather than assuming the key works. Count how often the receiver matches a key it has already completed. That number should be small and nonzero. Zero usually means the key is being regenerated per attempt and is not doing anything.
The pattern underneath all of this is the same one that shows up whenever automation handles its own failures: the system's report of what happened is not the same as what happened. A timeout is the case where the report goes missing entirely, and retry is what fills that silence with a guess.
Retries and idempotency are two halves of one guardrail, and the book works through the pair together. If you are adding retry to anything that writes, One Agent, One Company ($9.97) covers the keys and the duplicate checks that keep it from doubling your work.