Industry


Ads by TechWords

See your link here


Sharky's picture
Sharky

Shark Tank

There's caution, and then there's caution

A very large thunderstorm is headed for this company's data center -- an unusual event that management isn't quite ready for, says a pilot fish working there.

"When several similar companies merged in the late '90s, the one with the newest data center was chosen to be the production data center," fish explains. "However, it was originally built for a much smaller company than the merged entity, so it had to be immediately expanded.

"In the intervening years, the quantity of servers and network and disk hardware in the data center increased rapidly, but the UPS and emergency generator didn't keep up with the growth."

So with a big storm on its way, one high-level manager overrides the advice of the technical support team and decides to switch the data center over to emergency power, rather than risk the possibility that the city power could be disrupted.

The emergency generator is started up and the switchover is made on a Friday evening without a hitch.

But it turns out the generator has never been tested since it was installed -- and the diesel fuel it runs on had been sitting in the tank for a long time, so sludge has accumulated in the bottom of the tank.

When the sludge reaches the fuel line, the engine sputters to a stop. Shortly after that, the underpowered UPS runs out of juice.

And the entire data center shuts down catastrophically.

"The important servers and network were back up and running within 24 hours," sighs fish. "But some of the large databases took days to recover due to data corruption.

"And by the way, the city power never had an interruption."

The Shark Tank runs on your true tales of IT life. Send me your story at sharky@computerworld.com. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Now you can post your own stories of IT ridiculousness at Shark Bait. Join today and vent your IT frustrations to people who've been there, done that.

What People Are Saying

I'm really glad about all

I'm really glad about all the awesome stories on here. But why are all of you anonymous? We don't deal well with anonymity here.

Here is yet another story of

Here is yet another story of another moron, how do they keep getting into the data centers, they need to be shot on sight. And by the way, I had *my* name changed to Anonymous!

Actually, they're all my

Actually, they're all my comments. I had my name legally changed to Anonymous in the early 80's.

Was going to change it to O(+> but the artist formerly now again known as Prince beat me to it.

Follow up

I had a guy follow me up. It was kinda creepy. He said that I should do exactly what I would do at home, so I grabbed a few wafers of soylent green and tuned my neurovid to the Babmbi channel. He left a few minutes after that.

LOL!

MH needs to get his time machine adjusted. This post is for 9/14's tank.

The Babmbi Channel

It doesnt' get any better than that!

CAPTCHA: Knixks galosh - one must wear galoshes to get through some of the stuff the Knicks put on the court

ups's, transformers & diesel generators

This all happened right after Katrina hit New Orleans.

One of our transformers was leaking TCB's and needed to be replaced. Our technical staff decided that we would change over to a diesel generator 3 days before the change out of the transformer.

We tried several generators and even had one brought in from another city. During each change out the entire server room was running on the ups. One more thing, the new transformer was lost in the confusion following Katrina. (It was eventually located.)

Fast forward to Friday afternoon about 4:30. Our electricians realized that the ups did not like diesel generator power ((not the very clean power that was available through street power.) And we suddenly realized that the ups was not fully charged. Unfortunately, this realization occurred after we had been removed from the diesel generator and we were running completely on the ups.
(Remember, this is friday and everyone is saving their documents & finishing saving info to the databases.)
Suddenly, the server room was dead quiet, no power.

Four years later, all is fixed, right?

Nope, same problem, one more leaking transformer & the same ups's with more batteries.

When a Co-Gen Plant powers up...

...it needs cooling. Of course, when the cooling power is supplied by street power rather than itself, the co-gen plant shuts down rather quickly.
This happened when the Power Company had shut down our main power line to service it, and a local construction company building a new Convention Center hit the remaining one...
Our Facilities guys had to get creative to supply power to the cooling system by hot-wiring the output of a nearby diesel unit to the co-gen plant cooling system! In the meantime, we had four plus hours of very wacky power service as the Facilities guys attempted to patch through our other Diesels until they got the Co-Gen plant back up and running.

Ours was not so bad. We

Ours was not so bad. We were having our generator tested and the guy testing it went back to his truck to take a nap. The generator went wild and blew up. The company came in and told me they would have us a new one put back in place for about $12,000.00 or so. They came back 2 days later with a figure of $36,000.00+. Our board threw a fit because it was not something that we had done. Needless to say, we still do not have a generator but we have been very blessed not to need one. Thanks to back up plans and backup locations.

Then there was the Data

Then there was the Data Center with 3 Diesel Generators of which 2 could handle the load. A test was performed to see how well the cut-over worked.
The first generator started right up and took the computer load. The second started right up and took the cooling load. The first generator died. The third generator stated up to replace the first. The second generator died. The third generator could not power everything, but it died anyway.
Some observant person saw that each generator had run approximately 30 minutes. Upon further investigation, it was discovered that each generator had a small gravity feed fuel tank on top the would run each generator for about 30 minutes. Each of these tanks were fed by fuel pumps from a very large fuel tank.
Sounds like a good hook up till they found the fuel pumps hook up to the outside electrical supply. Catch 22 at its best.