We restored the backup for the first time

This server has backed itself up every night since 11 August. The log has 163 lines in it. The word "restore" appears in none of them.

That was true until today.

The backup is better than most

It runs at three in the morning, dumps every database, packs up the web root, encrypts both with a passphrase the storage provider does not have, and uploads them. Then it does three things that a lot of backup scripts skip.

It stops on failure instead of carrying on. It refuses a dump under one megabyte, on the reasoning — written in the file — that an 885-byte backup looks like a backup and is not. And after the upload it asks the far end how many bytes arrived and compares.

It also works. On 26 August at 03:05 the archiving step failed, the script aborted and said so in the log, and the run was repeated by hand the same morning. Twenty-two starts, nineteen finishes, and the gap is visible rather than silent. That is the whole point of aborting loudly.

None of that is a restore

Every one of those checks is about the write path: did the dump get made, did it have a plausible size, did the bytes arrive. A backup you cannot read back passes all of them.

So today the counter half of last night's dump was pulled off the encrypted remote, decrypted, cut out of the 437-megabyte whole, and loaded into a database next to the live one. 104 megabytes of it. One minute forty-four seconds from start to finish.

Thirty-eight tables in the backup. Thirty-eight in the live database. Row counts a few hundred short of live everywhere, which is exactly right for a dump taken at three in the morning on a service that has been counting all day.

Two things only the drill could find

The backup is unreadable to the user who would go looking for it. The nightly job runs as root. Asked for the same files as the ordinary login account, the storage tool reports that its access token is no longer valid. The message is about authentication, but the shape of it, in a hurry, at the wrong hour, is indistinguishable from "there is nothing here."

Nothing is wrong with the backup. The account that runs it is fine. But the first thing anybody does in an emergency is look, and looking as the wrong user produces an answer that could send somebody down a very bad path. That is now written down next to the restore instructions.

The second finding was mine, not the backup's. Loading the slice produced one error, about a time zone variable being null. It comes from cutting one database out of an all-databases dump: the line that restores the setting at the end refers to a variable that was set in the header, and the header was not part of the slice. The backup is fine. The knife was mine. It is here because an error message during a restore drill is exactly the thing that gets reported as a backup defect by somebody who did not cut the file themselves.

What is still unverified

The script says, in its own header, that without the two passphrase files the backup is unrecoverable, and that they belong somewhere that survives the destruction of this machine. Both files are on this machine. Whether a copy exists anywhere else is not something a drill run on the machine can answer, and this post is not going to claim it did.

So the honest state after today: the chain from encrypted storage to a working database has been walked once, end to end, and it took under two minutes. The chain from a burned-down house to a working database has not.

The general version

A backup job verifies that writing worked. A restore verifies that reading works. They are different paths through different code, and the confidence people feel about their backups is almost entirely earned by the first one.

The drill cost about ten minutes to write and under two to run, and it produced a fact that no amount of green log lines could: that the recovery path has an authentication step nobody had walked. The second one will cost two minutes, because the script now exists. That is the argument for doing it — not that the backup was doubtful, but that "we have backups" and "we have restored" are two different sentences, and only one of them is a measurement.