Shark Tank: Never assume

Flashback to the 1980s, when this sysadmin in charge of a minicomputer sets up a system of daily backups that should be just about bulletproof.

"I set up daily disk-to-disk backups with three sets of disks -- grandfather-father-son -- which could be done during lunch," says fish. "I also set up weekly tape backups that ran about four hours, and we kept six sets of tapes around before reusing them."

And he feels pretty secure about the safety of his data.

At least until a few months later, when a data-center operator discovers he's made a mistake and failed to swap disk packs before starting the second part of the daily backup routine.

Operator's solution? He restarts the system, which hoses both of the disk packs involved.

OK, says fish, that's why we have this system. We'll restore from the grandfather packs. We'll lose a day or two of data, but we'll be able to recover within a few days.

Um, no. Turns out the operator didn't see any reason for the three-deep rotation, and hasn't used the third set of disk packs for backups in three months.

Well, fish says, we still have the weekly tapes. We can restore the latest set, lose a few days' more data, but be recovered in a couple of weeks max.

Uh, no again. The operator noticed the backup tapes were never used, so he stopped running tape backups three months before, too.

"So our most recent backup was three months old," groans fish.

"After an outage that lasted one week instead of two hours, we managed to get back most of the lost data. And I learned a valuable lesson: Audit the procedures every now and then, because the operators may not be following them any more."

Submit your own true tales of IT life to sharky@computerworld.com. If Sharky uses it, you'll snag a snazzy Shark Tank shirt! You can also add comments by using the form at the bottom of this page.