Two bombshell papers released at the Usenix FAST '07 (File And Storage Technology) conference this week bring a welcome dose of reality to the basic building block of storage: the disk drive.

Together the two papers are 29 pages of dense computer science with lots of info on populations, statistical analysis, and related arcana. I recommend both papers. The following summary, and two longer analyses at StorageMojo are summaries of what I found interesting.

If you're retiring in the next five years you can skip this article. Otherwise, listen up.


RAID arrays were great in their day. But that day is drawing to a close. Managing LUNs and volumes, paying 20x the cost of the raw capacity for protection, poor scale-out: RAID arrays are just not competitive for large-scale infrastructures.




Storage clusters are now a proven commodity, with support from companies such as Oracle, IBM and NetApp. Highly resilient, simplified management, much lower cost. What's not to like?




Here are some examples:

• The world's largest data centers, Google, Amazon, Yahoo and Microsoft's MSN, all use storage clusters for 7x24 availability in their advertising operations
• At least a dozen more firms are selling cluster storage, including NetApp, the fastest growing large storage company
• Polyserve and Red Hat's GFS focus on storage clusters for Oracle and DB2 databases. With Oracle and IBM support.
• Omneon, a company specializing in storage and multi-media support for broadcasters, is selling "Media Grid" storage clusters. TV stations are a real-time 7x24 production environment: if the stuff doesn't work, the TV station doesn't get paid. It works.
• A storage cluster company - Isilon - just went public with a $1.4 Billion market cap

Arrays aren't going away tomorrow, or ever. It took 10 years from the publication of the Berkeley RAID paper before RAID arrays took 50% of the external storage market. Yet storage cluster use has been growing rapidly in some major niches: internet data centers, video and broadcasting and web services. The 85% of enterprise data that is unstructured is the next big market for storage clusters.

Recently, the always thoughtful CTO of Hitachi Data Systems, Hu Yoshida, had a couple of posts (here and here) that point up a dilemma. In one, a CIO is decrying the 20% utilization of his data center storage, and in the other Hu is marveling at the fact that storage is so cheap internet entrepreneurs can give it away for free.

So which is it: cheap enough to give away for free or costly enough to worry about low utilization? The answer, of course, is both. Data center storage is expensive, and the internet storage is cheap.

Let the fear-mongering begin: the Federal Rules of Civil Procedure (FRCP) requirements are changing effective December 1, 2006, with new requirements for electronic data discovery (EDD). Images of a pale and blinking CIO walking through phalanxes of shouting reporters and photographers may come to mind. But the reality is not so dire.

What is electronic data discovery?
In civil cases (non-criminal lawsuits mostly) judges want everyone prepared. The FRCP rules of discovery lay out the process by which both sides get to search each other's files looking for evidence that supports their case. It's an existing process extended to electronic information.

Google's Abject Failure


Google's $1.65 billion purchase of on-line storage provider YouTube is sending shock waves through the industry.

• Roelof Botha of Sequoia Capital, the hippest actuary in Silicon Valley and the only VC to invest in YouTube is saying "neener, neener" to Mark Cuban, John Battelle and all the others who just couldn't see how YouTube would ever make them money.
• In its 21-month life, YouTube has created value at the rate of $80 million dollars a month, sending every venture capitalist and smart kid searching for the "next big thing" in Web 2.0
• Founder Chad Hurley has proved that even a BA in Fine Arts can make you a billion dollars
• Finally, it is Google's first public admission of abject failure, failure that is costing them $1.65 billion and which will rend the carefully fostered geek culture of the company

Dateline: Silicon Valley
I went to the Datacenter Ventures conference last week to see what the cool kids are cooking up. There were over 60 presenting companies, ranging from tiny startups looking to raise their first VC dime to companies that have already raised tens of millions, have customers, and aren't looking for money.

Themes I liked: Google-style scalability; commodity hardware; datacenter automation; faster search; enterprise open source. Read on for specifics.

Rumors Lots of speculation about Cisco's Nuova subsidiary. Conventional Wisdom is that Nuova moves beyond the network into the hyper-scale cluster market. Since they own the network market they need to find another growth engine and it has to be something big.

One of the most misguided computer industry marketing initiatives ever is coming to a slow and shuddering halt. People just don't have time for nonsense. Sadly, Information Lifecycle Management (ILM) is particularly seductive nonsense. It has taken far too long for practitioners to see past the vendor's honeyed words.

As I noted over at StorageMojo.com, the good folks at StorageNetworking.org have raised a red flag over ILM in three reports: Information Lifecycle Management in Perspective: Initial Findings From Surveys of Top Management , Information Lifecycle Management: An Analysis of End User Perspectives and ILM Survey: What Storage, IT and Records Managers Say. Each is the product of their own independent research.

The storage industry is getting its share of attention lately from the feds with allegations, criminal charges, civil complaints, lawsuits and even the occasional guilty plea. Storage is a big business and for some the stakes are high enough to justify shady or even illegal tactics.

Did Rambus Rambo JEDEC?

The Federal Trade Commission, in a 5-0 vote, today issued an opinion that Rambus "unlawfully monopolized" markets in four memory technologies.

The FTC opinion detailed the Rambus villainy:

Coolest storage product of the year may be Sun's new X4500. Packing 24 TB of storage and 2 dual-core 64-bit Opterons into a 4U box, it is the first system anywhere to leverage the very cool open-source ZFS (see StorageMojo's overview ZFS: Threat Or Menace?). True end-to-end data integrity. RAID-Z performance. Cheap snapshots. 128-bit addressing. All backed up by the most brutal test suite I've ever seen.

ZFS: The First Internet-Scale File and Storage System

Much coolness over at HDS this week as they announced their OEM deal with Diligent Technologies. As I've noted at StorageMojo.com, I am a big fan of Diligent's 25x backup data compression technology that is part of their Virtual Tape Library product.

Fuzzy Logic, Meet Fuzzy Marketing
Oddly, these shy-and-retiring ex-EMC'ers don't call their product data compression. Yet doesn't something that reduces your backup data volume by 25x over time sound like a compression product?