Because when the unthinkable happens, it's a little late to start thinking

It's circa 1990, and this pilot fish works in IT operations in a big research university's data center.

"This meant our data center had both academic research timeshare systems and a mainframe for the administrative functions: payroll, student registrations, library systems, etc.," says fish.

"Since we were a 24/7 operation, we had redundant power feeds and a UPS designed to hold up the entire site for 20 to 30 minutes -- usually more than long enough to switch between the power feeds -- but no generator."

That also means that the data center staff has never had to shut everything down and go dark.

Then one night the local power substation takes a lightning strike, and the entire power grid for half the county goes offline -- including the campus.

It takes a few minutes for fish and his co-workers to realize they've lost both their power feeds. Then it takes another minute or so to realize that they have to shut everything down -- and fast, before the building UPS dies.

Fish is tasked with shutting down the mainframe. "Following the written documentation for emergency shutdowns, I start taking peripheral devices offline, then try to shut down the OS," he says.

"To my surprise, the system started prompting to have tapes mounted. It seems that the admins for that system had all the shutdown procedures designed to back up the system disks before taking the OS down."

Fish repeatedly cancels the tape-mounting process and tries to force the shutdown. But each time he does that, the system again tries to mount tapes on a drive that fish has already taken offline.

He manages to get a mainframe admin on the phone. After walking fish through a few things that don't work, all the admin can suggest is that fish bring one tape drive back online and let it try to back up.

Fish explains in heated terms that the nightly system backup takes at least 45 minutes -- and there's nothing like that much time left on the UPS.

"While we were going back and forth trying to come up with a solution, the UPS finally gave out and the mainframe went down hard," says fish.

"In the debriefing session we had the next day after power had been restored and all the pieces had been picked up, it was decided that a special command file would be put on the mainframe system to shut it down without requiring backup -- to be used in case of emergency."

Think of Sharky right now and send me your true tale of IT life at sharky@computerworld.com. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Now you can post your own stories of IT ridiculousness at Shark Bait. Join today and vent your IT frustrations to people who've been there, done that.