As predicted, sort of

It's the weekend of a big system upgrade for this not-so-big company, and everyone wants to make sure everything goes well, says a pilot fish on the scene.

"We hired an outside consultant who knew the old system and new system very well and was confident everything would go off without a hitch," fish says. "He even had tickets to head back home that afternoon.

"We tested the upgrade on the test box. All went well, although we weren't able to test the import function since there was no test environment set up for that."

Then it's the night before go-live. Fish's team takes all the necessary precautions, starts the backup early and verifies that the backups work. So far, so good.

The actual upgrade takes about two hours, and everything still seems to be going well -- even the untested import functions are working. Everyone breathes a sigh of relief.

"We had asked that several users come in for about an hour to test the system and make sure nothing went wrong," says fish. "All was going well for the first 15 minutes.

"Then someone popped up out of his cubicle saying 'I can't access the system.'"

Fish checks. The system that the user can't access isn't the one that was upgraded, but interacts with the upgrade. What's going on?

Baffled fish heads for the server room -- where he's greeted with the aroma of burning electronics. Turns out one of the UPSes has shorted out internally, has blown its fuse and is tripping a circuit breaker.

Fish knows most of the servers have redundant power supplies that feed off two separate UPSes, so there's no single point of UPS failure. And this one?

"Turns out the system that went down was one of only two systems that didn't have a redundant power supply," fish sighs. "And it was plugged into the UPS that failed.

"We restored power and ordered a new UPS and redundant power supply, and the rest of the upgrade went without a hitch -- just as the consultant predicted."

Sharky predicts that you've got your own true tale of IT life. Send it to me at sharky@computerworld.com. You'll get a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Now you can post your own stories of IT ridiculousness at Shark Bait. Join today and vent your IT frustrations to people who've been there, done that.