Danger! Microsoft loses T-Mobile Sidekick/Hiptop users' data

Microsoft has lost all users' personal information for the T-Mobile Sidekick service (aka Danger Hiptop). It's just gone, with little hope of recovery. Looks like there was a disk failure in its Danger subsidiary, and no working backups. In IT Blogwatch, bloggers don't know whether to laugh or cry.

By Richi Jennings. October 12, 2009.
(MSFT) (DT) (ETR:DTE)

Your humble blogwatcher selected these bloggy morsels for your enjoyment. Not to mention cutting up old VHS tapes...
 
 
John Paczkowski pokes fun at Microsoft:

In the canon of Microsoft [mis-steps], this may be the most humiliating: A server failure at the company’s Danger subsidiary has wiped out the personal data of a large number of T-Mobile Sidekick users. ... The Sidekick stores contacts, calendar entries, and other key data primarily on Danger’s servers, not locally. That’s a fine strategy when the information backed up in multiple redundancy RAID configurations.
...
Microsoft hasn’t yet said what caused the failure. ... Nor has the company said why it doesn’t have a copy of Sidekick user data. ... It’s hard to imagine ... a reasonable explanation ...  for unrecoverable data loss. ... This is an ugly embarrassment for Danger and Microsoft and one that will probably cost them the trust of Sidekick users. Sadly, Danger seems to have lived up to its name.

Om Malik likens it to a 3270:

Danger’s service works in a very simple fashion. The devices are in constant communication with a server which does everything from checking email to fetching web pages and maintaining contact with all the folks we know on [IM]. It also keeps copies of other communications (such as text messages), address books and calendars. It stores photos on its servers as well.
...
What we have is a device that is a combination of a cell phone and an almost dumb terminal.

Jason Kincaid offers a crumb of comfort:

That means that any contacts, photos, calendars, or to-do lists that haven’t been locally backed up are gone. Apparently if you don’t turn off your Sidekick ... you can salvage what’s currently stored on the device ... Microsoft/Danger is describing the likelihood of recovering the data from their servers as “extremely low”.
...
This goes beyond FAIL, face-palm, or any of the other internet memes we’ve come to associate with incompetence. The fact that T-Mobile and/or Microsoft Danger don’t have a redundant backup is simply inexcusable. ... Microsoft acquired Danger for $500 million in February 2008.

Seth Weintraub commiserates:

Customers will be happy to know that Tmobile is offering a free month of data (not a free month of service, just the $20 unlimited data plan) for all of their information. I really hope a lot of Sidekick users used the Intellisync software that pushes data to the desktop and would have backed their data up.

T-Mobile has halted the sale of new Sidekicks. All models are now showing "temporarily out of stock" on the company's website.

Ed Hunsinger asks, "How did this happen?"

Currently the rumor with the most weight is as follows: Microsoft was upgrading their SAN ... and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. ... Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in it’s destruction.
...
We’ve heard this from what appears to be several sources and it seems to hold weight. Needless to say it all boils down to one thing: Microsoft did not have a working backup. ... The head of the mobile division (and person in charge of what’s left of Danger) is Roz Ho, who has been at Microsoft for 18 years. You would think she’d know something about how to run a business.

Chris Ziegler ponders the impact for the carrier:

The coming weeks are going to be trying times for ... T-Mobile, a sideline player in this carnage that ultimately still shoulders responsibility for taking users' cash month after month. ... We're betting that heads are going to roll at both of these companies, formal investigations are going to be waged, users are going to be compensated in big ways, lawsuits are going to be filed, and textbooks could very well be modified to make sure that lessons are learned.
...
Why there weren't any backups -- even older ones -- that could've been used as a restore point is totally unclear, so we're hoping Microsoft has the stones to come clean for the benefit of an entire industry that wants to understand how to make sure this never happens again.

Tim Anderson asks, "What next?"

People are drawing a variety of conclusions, the most obvious being either that the cloud can never be trusted, and/or that Microsoft can never be trusted. Of course there is no such thing as total data (or any other kind of) security, but risks can be minimized, and in the absence of nuclear war, earthquake or volcanic eruption this looks inexcusable.
...
The company is promising an update [Monday]. Personally I doubt that the data is really irrecoverable, knowing a little about what data forensics can achieve, but it may be economically irrecoverable. Still, the best thing Microsoft could do would be to announce that it can get the data back after all. Failing that, we need to understand as much as possible about what went wrong so that we can make our own judgment about what to conclude.

So what's your take?
Get involved: leave a comment.
 
 
And finally...

• 'The Golden Age of Video' by Ricardo Autobahn
  [hat tip: b3ta]

Richi Jennings is an independent analyst/consultant, specializing in blogging, email, and security. A cross-functional IT geek since 1985, he is also an analyst at Ferris Research. You can follow him as @richi on Twitter, or richij on FriendFeed, pretend to be richij's friend on Facebook, or just use good old email: itblogwatch@richij.com.

 
 
Don't miss out on IT Blogwatch:

• Subscribe to the Computerworld Blogs and IT Blogwatch newsletters
• Catch up with posts from the previous few days