Failure to launch

Flashback to 1997, when this IT contractor pilot fish is working on a project for a U.S. government agency that he prefers not to name.

"But here's a hint: They put boot prints on the moon," says fish. "I was helping to develop a next-generation digital library and archive system for a new satellite observation system.

"The agency is rightly proud of this facility, which includes supercomputers, high-density storage systems capable of holding more information than all of human knowledge prior to 1950 -- and they have two of them! -- and a state-of-the-art power system complete with dedicated UPS and a large generator capable of powering a small town."

Because the generator is new, it naturally needs to be broken in, so over the course of the summer it's tested at least weekly. That means cutting off the main power feeds for the server room at the circuit breaker box.

Every time, over the course of three months, the generator performs flawlessly, producing full power within two minutes -- well within the 10-to-15-minute UPS timeframe.

Then one day a young woman loses control of her car three miles away and knocks out a major power line, taking the entire facility's power down.

"And nothing happens," fish says. "After two minutes without the generator starting to fire up, folks start scratching their heads. It was tested the day before and worked perfectly. The diesel fuel is at over 200 gallons. Nothing is wrong -- but the generator isn't starting.

"As people start scrambling to shut down the supercomputers cleanly, others are running to the generator room, but it's too late. Eight minutes in, the first systems crash hard, and within a minute, the last goes down."

Since there's nothing left to do in the server room, everyone gathers at the door to the generator room. The dim self-powered emergency lights show that the controller -- the PC that's supposed to tell the generator to start cranking -- is down.

In fact, it looks like it went down the moment power was lost. As the techs carefully begin to troubleshoot it, someone points to the power cord. It's plugged neatly into a wall outlet -- one that's on a different circuit from the server room.

Which means for every test over the past three months, when the breaker was switched off for the server room to mimic a power outage, it didn't interrupt the power to the generator room or the controller PC.

"Oops," says fish. "The next day, a brand new off-the-shelf UPS from a local hobbyist computer store was sitting beside the controller PC."

Got a disaster? Tell Sharky about it. Send me your true tale of It life at sharky@computerworld.com. You'll snag a snazzy Shark shirt if I use it. Comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.

Get Sharky's outtakes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.

5 power user tips for Microsoft OneNote
  
Shop Tech Products at Amazon