This is a story about being wrong in the same way twice in eighteen hours. The first time I caught it before it cost me anything. The second time I caught it because I went looking, and it had already cost me a day of backups without saying a word.
The first one
I was about to delete an old tree of roughly 54,000 files after migrating everything to a new location. Before deleting anything that size, I ran an audit across every project, then had three independent agents try to find where the audit was wrong. They overturned conclusions in both directions, which is the entire point of asking.
One finding stopped the deletion. The existing backup archive held 1,515 files. The live tree held 5,269. Two whole subtrees, about 3,700 files, were not in the archive at all.
The archive had a completeness check. It had been passing for weeks.
Here is why it passed. The check walked the archive, walked the inclusion scope, and compared the two. But the inclusion scope was the same list the archive was built from. The check was asking "did I copy the things I decided to copy," which is true by construction, rather than "did I copy the things that exist." When the scope silently lost two directories, both the archive and the check lost them together, in agreement, forever.
A check that derives its expectation from the same source as the thing it is checking is not a check. It is a mirror.
I rebuilt the backup system that afternoon. Better structure, credential exclusion enforced two independent ways, three scheduled tasks, four destinations.
The second one
The next morning I went to confirm the new system had run overnight. It had. The log was clean:
01:00:02 [Daily] ===== start =====
01:14:02 [Daily] archive written: cdev-2026-07-26.tar.gz (2981.3 MB)
01:14:05 [Daily] coverage: archive=0 expected=5486 secretsLeaked=0
01:14:05 [Daily] copied to M:
Read the third line again. archive=0. The coverage check listed zero files inside a 2,981 MB archive and passed anyway.
It passed because the only condition that could fail it was a leaked credential appearing in the file list. The file list was empty. An empty list contains no credentials. Pass.
Read the fourth line too. A three gigabyte copy did not complete in three seconds. It did not complete at all. Every destination still held the previous day's archive, and had for over 24 hours, while three scheduled tasks reported success each night.
Three separate mechanisms all agreed everything was fine:
- The copy step ran with error handling set to continue, no post copy existence check, and an unconditional log line announcing success immediately after.
- The coverage check logged the entry count and then returned a verdict that never consulted it.
- The script had no exit statement anywhere, so the task scheduler recorded success structurally, regardless of what happened inside.
Same defect class as the day before. Not the same code, not the same file, not even the same language construct. The same shape: a verification whose failure condition could not be reached.
And there was a nastier detail. The archive was a plausible size. It passed a file existence check and a non-zero exit check. Every cheap sanity test you would reach for first would have said it was fine. The only assertion that catches it is the one nobody writes, which is on the number of things actually inside.
What the check does now
It fails on five named conditions, and I wrote them down before I wrote the function:
- the archive file is missing
- listing the archive exits non-zero
- the listing returns zero files while the file is megabytes in size
- the listing returns fewer than 95 percent of the expected count
- any excluded credential file appears inside
Plus the plumbing that should have been there the first time: copies that stop on error and are checked for existence afterward, an exit code that reflects the outcome, and a file that appears on my desktop named BACKUP PROBLEM.txt when anything trips. Silence is no longer evidence of success.
All four destinations now carry same day archives. I verified that by listing them rather than by reading a log, which is the whole lesson in one sentence.
The rule
A check that cannot return false is not a check. Before writing a guard, name the condition that trips it. If you cannot name one, you are writing a mirror.
I had that rule written down on the 25th, in the document describing the first bug. I broke it on the 26th anyway, at one in the morning, in a system whose entire purpose was to be trustworthy. Writing the rule down is not the same as building a machine that enforces it, and the gap between those two things is where most incidents live.