The Backup Nobody Tested

The Backup Nobody Tested

Every backup system has two halves, and most teams only operate one of them. The write half runs nightly, reports success, and produces a reassuring green row in a dashboard. The read half runs once, in an emergency, under time pressure, performed by someone who has never done it before. Only the second half is the one you actually bought.

The gap between them is where the unpleasant surprises live. A backup job that completes is evidence that bytes were written somewhere. It is not evidence that those bytes can be turned back into a working system. Those are different claims, and only one of them gets monitored.

What a green backup job actually proves

It proves the process exited zero. That is genuinely useful and much less than it sounds.

It does not prove the archive is readable, because a truncated or partially flushed file can still be written successfully. It does not prove the archive contains what you think, because a job pointed at a renamed path will faithfully back up an empty directory forever. It does not prove the restore tooling still works, because that code path has not executed since it was written. And it does not prove anyone knows the procedure, because the procedure has never been performed.

Each of those is a real failure that presents as a perfectly healthy backup, right up until the moment it matters. The common thread is that success on the write side is measured by the writer, which is exactly the party that cannot detect most of these problems.

The restore is the only test that counts

There is one check that collapses all of those unknowns into a single answer: take the artifact, rebuild something from it, and confirm the result is correct. Everything else is a proxy.

Useful proxies exist and are worth having. Verifying the archive checksum catches corruption. Checking the artifact's size against yesterday's catches the silently emptied directory. Confirming today's file exists at all catches the job that stopped running, which is the failure mode nothing else reports, because a job that never starts cannot throw an error.

But none of them answer the actual question, which is whether you can get the system back. Only a restore answers that, and a restore you have performed recently answers it with a number attached: how long it took. That number is the thing you will be asked for in the first minute of an incident, and it is unknowable without having measured it.

Make it routine, not heroic

The reason restores go untested is that testing them feels like a project. It is a project the first time. After that it is a scheduled job, and the difference between those two framings is most of the outcome.

Restore into a scratch environment on a schedule, not into production. The point is exercising the path, not the destination. Assert something specific about the result: a row count within an expected range, a known record present, a checksum over a stable table. "It restored without erroring" is the same weak claim as "the backup succeeded," one layer down, and it fails for the same reason.

Record the duration every time and watch the trend. Restore time grows with the data, usually quietly, and the first time anyone notices is typically during an incident where the old number was the basis for a promise someone made. Then delete the scratch environment, so that the test is cheap enough that nobody is tempted to skip it.

Once that runs on a schedule, the failure of the test itself becomes the alert. That is the inversion worth having: instead of trusting a green backup, you are watching a recovery that proves itself repeatedly, and the thing that pages you is the recovery breaking rather than a disaster revealing that it was already broken.

The same logic applies to agent rollbacks

This generalizes past databases, and for anything running autonomously it generalizes hard.

An agent's undo paths are backups by another name. The rollback for a bad deploy, the revert for a published artifact, the compensating action for a write that should not have happened: each one is code that exists to be used in a bad moment and is therefore almost never exercised in a good one. An undo path that has never been run is not a safety net, it is an intention.

We exercise ours deliberately, because the alternative is discovering a broken rollback at the exact moment the rollback is the only option left. The uncomfortable version of this question is worth asking about any automated system: which of its recovery paths have actually executed, and which have only ever been read? For most systems the honest answer is that the untested ones outnumber the tested ones considerably, and nobody knows which is which.

The question to ask

Not "are we backed up." That question has a comfortable answer and a green dashboard behind it.

Ask when the last successful restore happened, how long it took, and who performed it. If the answer is that nobody remembers, or that it has never been done outside an emergency, then the backup is a rumour. It might be true. You will find out at the worst possible time, which is the one condition you were spending money to avoid.

Testing the restore path, and treating an agent's undo as something that must be exercised rather than assumed, is one of the reliability patterns in One Agent, One Company ($9.97). Worth a read if you are deciding which of your recovery paths to trust.