To Pluto and Beyond!

Thanks to EMC’s sponsorship, the effort started by the UC Berkeley School of information in 2003 to quantify the global trends in creation of digital data continues under IDC’s able supervision. The latest report, “The Digital Universe Decade,” is hot off the press, and breaks the banks on crazy metaphors for how much data is produced and consumed. They have the usual “moon-and-back” metaphor, but here’s my favorite: 707 trillion copies of the 2000+ page U.S. Healthcare bill stretched from Earth to Pluto and back 16-times. Nothing like a little dig at congress while we’re at it!

The most interesting trends from our perspective were:

1) The growth in unstructured data continues to outpace structured data,

2) Thanks largely to widespread media-enabled cell phone use, 70% of data will be user-generated

3) This one really jumps out: 35% more digital information is created today than the capacity exists to store it. This number will jump to over 60% over the next several years.

So what the IDC numbers mean is that platter density can’t keep pace with the market. Although John Toigo (http://www.drunkendata.com/?p=2872) refutes that by arguing platter densities have kept ahead of IDC’s expectations, I expect the truth to be somewhere in the middle. Either way, the ability to find more disk and datacenter to store the deluge is going to present more challenges for Sysadmins, who are already up to their necks in data.

Good news for vendors who supply storage hardware. No surprise it’s EMC who sponsors this study!

Administrative issues aside, it raises a question of IT sustainability. The IT budget in a typical Fortune 1000 enterprise needs to track as a percentage of overall capex and opex. When IT in a typical enterprise must grow as a % of overall expenses, that can’t continue; IT has to consider itself non-scalable. With the scenario described in the report, IT organizations are going to encounter this exact scenario. Their storage budget will grow as a percentage of opex and capex, and that can’t continue indefinitely.

So the market has two choices:

1) Throw away data.

2) Greatly improve utilization, through technologies like thin provisioning and data reduction.

History has already shown that throwing away is not going to happen because the IT staff managing it doesn’t own the data, and they have neither the tools nor the buy-in to begin expiring the data of their users. So primary storage optimization is getting a ton of attention. We anticipate data reduction features rapidly moving from “should have” to a “must have” in storage solutions globally, and enterprises globally are being encouraged to start evaluating reduction technology if they want to keep storage budgets in check. In fact we’re starting to see that sentiment in F1000 polling by TheInfoPro (www.theinfopro.com): Their two latest survey waves are showing Online Data Reduction as ranked 2 of 21 in their Storage Networking Technologies Heat Index.

In a perfect world hardware would be free and wouldn’t generate heat, users would diligently delete files, the IT staff would triple and we’d all win the lottery, which would allow you to safely store the 1.2 Zettabytes of data that’s crossing your LAN in 2010! But the boss says we shouldn’t count on those things, so we’re working hard on end-to-end solutions for a more intelligent way to deal with the data explosion.