Delivering the now: How the data deluge is driving innovation

With the rise of social media outlets like Facebook and Twitter, we have started to take over the role as both consumers and creators of data. However, the Internet, web servers and web application servers were initially designed for the mid-1990s Internet scenario to allow people to access information quickly rather than create the information themselves. As we increasingly take over the role as publishers, instead of consumers, the architecture of these web services is not keeping up with the data demands as we increase our number of tweets, photos, et al.

The problem with delivering "the now"

The issue is that our expectation is that the posting we just made appears instantly, letting our friends know in "real-time" what we are up to through a status update or picture, but these systems were not designed with this in mind. Developers of these web servers and other systems were not expecting the high level of data that would be pushed every second of every minute of every hour. Additionally, the components that make up the bulk of the web such as servers, caches, web/app server and programming languages were not designed for what we're asking them to do in terms of user data creation.

An emerging class of Internet infrastructure technologies

In order to address this problem storage systems need to be completely overhauled. The software layers that comprise web and application servers need to: a) be prepared for the large volumes of data that is being pushed from individuals into storage on these public websites such that the end users are not waiting, and b) have it done in such a way that these software systems are scalable so that the owners of these websites do not experience a dramatic increase in spending to support this massive amount of data.

The data paths for technologies like programming languages, application frameworks and web servers need to be optimized for the writing of data and we are starting to see increased effort around this particular approach. Node.js is one interesting example. Node.js designers observed that blocking I/O killed performance in heavy write scenarios. So they designed in non-blocking I/O among other things. They didn't invent non-blocking I/O, but rather applied to the problem at hand.

The hardware folks are not standing aside and watching. The notion of packing servers into datacenters to run the new applications on machines closer together gives us better power, cooling, and hardware subsystems designed to get the new job done. IOPS is suddenly a critical selling factor for products. Newly public company, Fusion I/O, is a well known example of a company that came into existence around this need. The guys at Isilon Systems were bought by EMC for a huge multiple for their technology.

A few companies in stealth have come to the realization that these technologies developed in the 1990s are discombobulated (programming languages don't fit next to the designs of the web server). As a response they are attempting to redo and collapse all of these inefficiencies between these subsystems and make them much more scalable to address the data deluge.

In conclusion, the need to deliver the now is being met with truly innovative companies and it will be exciting to see what happens!

Frank Artale is a partner at Ignition, a venture capital firm, where he specializes in cloud, core infrastructure, networking, and security investments. Follow Frank on Twitter: @frankartale. 

Neither Frank nor Ignition has any financial interest in any company or product mentioned in this post.