Why Thomas Jefferson could predict the next big thing in storage

In 1814, Thomas Jefferson donated the contents of his vast personal library of books and correspondence to form the foundation of the Library of Congress. Some 200 years later, that library is one of the largest in the world. Yet, the text of all of its contents could fit on a stack of DVDs that would reach to the top of a two-story building. Meanwhile, as I mentioned in a previous post, the amount of unstructured data created every year could fill a stack of DVDs from here to the moon and back.

Throughout this series of blogs, I have argued that the explosion of data, and trends such as virtualization, big data, and the cloud, are driving fundamental changes in the storage industry. I have argued that  monolithic, "big box," hardware-bound storage architectures must inevitably yield to approaches that deliver storage as a virtualized, commoditized, scale out pool on lots of "little boxes." In such a model, the value (and the solution for achieving high reliability and performance) comes in the software, rather than the hardware.

In this post, I will peel back the onion one level further and state that the software that will dominate storage will be open sourced.

My prediction that open source will dominate storage is partially based on comparative history. Data center computing has moved dramatically away from Solaris on RISC to Linux on x86. Indeed when looking at the entire computing stack, fully open source approaches like LAMP (Linux, Apache, MySQL, PHP) have proven irresistible. That trend is seen dramatically across multiple parts of the IT stack. The graph below from Netcraft shows the growth of Apache on web servers, but similar trends can be seen in server operating systems, database, etc.

© Netcraft Ltd 2011

To some extent, this is a matter of simple economics. Not only is a copy of an open source piece of software such as Linux generally far less expensive than a proprietary counterpart such as Solaris, it also opens up value throughout the entire stack. LAMP, for example, opened up superior economics in application development, database, operating systems, and the server hardware itself.

However, a deeper reason behind this belief in the inevitability of open source storage comes from a belief in the inherent power of openness to flourish in times of great technological change; an idea that Thomas Jefferson himself endorsed in a letter he wrote in 1813 (see an excerpt below). It is no surprise that the LAMP stack is open. In a time of explosive change (e.g. Web 1.0), it would have been nearly impossible for any single company to get all of the technology right or produce a single, comprehensive stack. By having all layers of the stack open, it was possible for many different organizations, both commercial and non-commercial, to create well-integrated programs that worked well together and leveraged the advances being made across the technology spectrum.

Today, we are facing massive changes from multiple directions, which are far more dramatic than those seen in the late 1990s. These include the move to cloud architectures and the emergence of the big data/SMAQ stack. Storage plays a pivotal role in all of these changes. If the storage industry is to both support and draft off of the new directions in IT, it must become more open. If the current crop of storage companies try to keep storage closed in an era of Hadoop, Pig, Hive, OpenStack, et. al., they will find themselves left behind.

As Jefferson said in an 1813 letter (included in that initial donation to the Library of Congress):

If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it. Its peculiar character, too, is that no one possesses the less, because every other possesses the whole of it. He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me. That ideas should freely spread from one to another over the globe, for the moral and mutual instruction of man, and improvement of his condition, seems to have been peculiarly and benevolently designed by nature, when she made them, like fire, expansible over all space, without lessening their density in any point, and like the air in which we breathe, move, and have our physical being, incapable of confinement or exclusive appropriation.

Ben Golub was CEO of Gluster, Inc. , which is now the Storage Business Unit of Red Hat. He is on Twitter @golubbe.