Shark Tank: What could go wro--

This pilot fish runs the IT department at a small rural hospital, and that means lots of challenges. But keeping the power coming isn't one of them -- right?

"Of course our servers are plugged into a UPS, and that is plugged into circuits powered by the backup generator," fish says. "In the event of a power outage, the UPS only has to power the server for, at most, eight seconds until the generator power takes over.

"So, theoretically, for my servers to lose power, not only would we have to suffer an extended power outage, but the backup generator would have to fail to come online long enough to drain the UPS, which would have to be something in excess of 10 minutes, more likely an hour."So fish is surprised to arrive at work and find lots of voice messages telling him the hospital's main application was down between 6:30 and 6:50 a.m. A quick review of the server logs shows they rebooted twice during that 20 minutes.

But why? If there was a power outage and the generator failed to kick in, all the lights would have gone out, not just the servers, and fish would have been called in from home."I called the emergency room clerk because I knew she would have been here in that time frame," says fish. "She reported that the lights flickered at 6:30 but there were no power outages."

Next, fish checks with the plant maintenance supervisor, who confirms that, at 6:30 a.m., he fired up the backup generator for its regular weekly test. It came online properly, which explains the flickering lights.

That makes the server UPS a prime suspect. But it looks fine: Indicator lights say the battery is fully charged, load is OK and nothing's wrong."So, without thinking of the consequences, I pushed the test button on the UPS," fish says. "And I watched in horror as the UPS went dark and my servers went down for the third time in just a few hours.

"Of course, now it was after 8 a.m. and many users were in the system -- many upset users.

"We spent the next 90 minutes putting the servers on spare UPSs and working with our main application vendor to repair files and directories that were corrupted during the hard crash as the phone rang off the hook with user complaints.

"Needless to say, I will be devising procedures to test server UPSs during any and every scheduled downtime."

Submit your own true tales of IT life to sharky@computerworld.com. If Sharky uses it, you'll snag a snazzy Shark Tank shirt! You can also add comments by using the form at the bottom of this page.