365 Main's mains power outage outrage (and popcorn)
2 comments- IT TOPICS:Hardware, Security, Servers & Data Center, Software, Internet
That'll be Thursday's IT Blogwatch: in which the power in goes out in SF and our favorite Web sites go too. Not to mention fake movie posters and real movie trailers...
Robert Mullins mulled over the irony:
A power outage in parts of downtown San Francisco yesterday affected office computers, lights and a hosting company for several Web sites. Pacific Gas & Electric Co., the local utility, reported 30,000 customers were without power as a result of an explosion under a manhole cover on Mission St., according to the Web site of the San Francisco Chronicle. The outage lasted from about 1:50 p.m. to 3:00 p.m., Pacific time.Among the businesses affected was 365 Main Inc., a Web hosting company at that San Francisco address. Among the Web sites it hosts is CraigsList.com, the popular global community Web site. An error message greeted people trying to get to CraigsList.com, although that site is back up, a CraigsList Inc. spokeswoman said. The 365 Main facility also serves Yelp.com, a user-generated review site focusing on restaurants, travel and other subjects.
The outage also darkened the offices of Computerworld's corporate parent, International Data Group Inc. The IDG facilities include the offices of PC World and Macworld. [more]
Frank "+++ATH" Hayes adds:
So much for the fantasy of the bulletproof Web site ... 365 Main's official position is that they went to backup power, but that doesn't match up with what SixApart (the TypePad and LiveJournal people) was reporting through its support site. Craigslist was also offline, as were the other sites at 365 Main. Gift retailer RedEnvelope had just shut down a backup site in the midwest after two years without a glitch at the main San Francisco site ... No doubt a lot of questions are being asked.But it's yet another wakeup call for businesses that rely on the Internet (and that's getting to be just about all of them): Co-hosting isn't perfect. Backups need backups. Contingency plans need contingency plans. [more]
IBM's Todd Watson (no relation) calls it a wake-up call: [sigh -Ed.]
Though it reported that some 30 to 50K customers of Pacific Gas and Electrics Co. were impacted by the power outage, I would submit to you that it's more likely that hundreds of thousands and perhaps millions of customers around the globe were unable to get access to any number of Web businesses.
...
Many Webizens turned to Twitter as the redundant and dependable back channel of communications choice: "Hey dude, your Web site's down and you're losing thousands of dollars in orders." ... I expect many of those businesses being hosted at 365 Main will be quickly revisiting their service level agreements. [more]
Owen Thomas wags the dog:
A source close to the company says: "Someone came in [very] drunk, got angry, went berserk, and [messed] up a lot of stuff. There's an outage on 40 or so racks at minimum." ... We're sure 365 Main will deny that such a thing could ever happen. And, conveniently, the neighborhood is having power troubles, too. But here's a question: When you have several levels of redundant power, what could bring your customers' servers down other than something like an employee ... hitting the emergency-power-off switch that San Francisco's building codes require 365 Main install? ... This isn't the first time 365 Main has suffered an EPO-induced outage; a major one still remembered by customers occurred back in April 2005, and another took place last year.
...
Sightings ... A man being lead away by police, in handcuffs, screaming, "You have been trolled by nut rollers!" Could this have been the employee responsible for the outage?
...
Cynthia Harris, the same flack who issued an immaculately timed press release Tuesday morning crowing about how RedEnvelope moved all of its Web operations to 365 Main, only to have them taken down by the outage, is going around telling everyone who will listen that nothing untoward happened. To which any user of Craigslist, Technorati, Six Apart's LiveJournal and TypePad, and AdBrite might respond, rrrrright. [more]
To which, SOMAFM responded:
365main ... have these things called "CPS", or continuious power systems. What they are is very very large flywheels that sit between electric motors and generators ... The flywheels ... can run the generator at full load for up to 60 seconds according to the specs ... There are also 10 large diesel engines up on the roof as well, connected to these flywheels. If the power is out for more than 15 seconds, the generators start up, and clutch in and drive the flywheels.Here's what I think happened. Since there were several brief outages in a row before the power went out for good ... the [flywheels] weren't fully back up to speed when the next outage occurred [but] the generators were not automatically started ... By the 6th power glitch, there wasn't enough energy stored in the flywheels to keep the system going long enough for the diesel generators to start up and come to speed before switching over. [more]
But Robert Scoble scoffs:
Ahh, Valleywag demonstrates one more time that they’ll print any damn thing sent to them without any care whatsoever about whether or not such a thing is true or not.
...
I gotta get myself a fake email address so I can send [stuff] into Valleywag and get it printed. Jason Calacanis told me about how he got free publicity for Mahalo that way. Anyway, I love Valleywag. The stuff that people get printed there is *funny.* Just don’t confuse it for the truth.. [more]
Jeffrey Baker leaps to Thomas's defense:
365 Main has a long and ignominious history of frequent and prolonged power outages, yet it remains fully booked. Some people just can't learn a lesson. [more]
Om Malik winks and says goodbye:
Web’s Weakest Link: The Power Grid ... yesterday was a rude reminder of how fragile our digital lives are. The seemingly invincible web services (not to mention the notional wealth they signify) vanish within a blink of the eye. It was also a reminder, that all the hoopla around web services is just noise - for in the end the hardware, the plumbing, the pipes and more importantly, the power grid is the real show.According to North American Electric Reliability Council (NERC) there has been a 50% decline in the capital expenditures by utilities over the last 15 years. The underground cables are crumbling ... It is ironic because data centers/hosting business is one where US companies still dominate. Thanks to an abundance of long haul bandwidth and ample data center capacity, many overseas companies find it more secure to host their digital infrastructure in the US. Bebo might be big in Britain, but its websites are served up from 365 Main. [more]
Buffer overflow:
Around the Net
- Richi Jennings: Much Love to robtex.com
- Data Center Knowledge: Colo Providers Partner on Disaster Recovery
- Larry Dignan: Esther Dyson is posting her genome on the Web; Will you follow?
- Stephen Wildstrom: iPhone: Analysts and Reality
- The Old New Thing : What is Dave's frame class?
- Joel Hruska: Report: Seagate plans to stop manufacturing IDE drives by year end
- IT Managers Journal: Forever Chasing Disk Space
- Inside Microsoft: Windows Genuine Advantage Customers Play Role In Piracy Bust
- Pareto's Practical Paradox of Project Management
Around Computerworld
- David Ramel: Magazines offer opposite takes on Second Life
- Mitch Betts: IT departments can use a blog to communicate with their user community
- Lucas Mearian: Robbers even prefer Blu-ray over HD DVD
- Joyce Carpenter: Tracking the source
- Michael R. Farnum: MySpace gives the boot to 29,000 sex offenders
- Shark Tank: No. Really. Down.
- Douglas Schweitzer: Let's lose "www"
- Shark Bait: Zune or iPod?
Previously in IT Blogwatch
And finally... The Indelible Hulk, Harry Potter and the Order, Trailers From Hell
Richi Jennings is an independent technology and marketing consultant, specializing in email, blogging, Linux, and computer security. A 20 year, cross-functional IT veteran, he is also an analyst at Ferris Research. Contact Richi at blogwatch@richi.co.uk.
Print
Email this
Digg this
Slashdot this
