Online Storage Optimization

With all the talk about the data inconsistencies around climate change theory, one issue that I’d hate to see lost in the shuffle is the actual environment. That is, while I personally have been skeptical for some time about the alarmist tone many scientists took regarding global warming, it would be a shame if there was such a backlash that people forget about the much more crucial, larger issue at stake. That is, we need to look at all the ways –on macro- and micro-scales–that we can reduce the overall pollution we generate through our daily habits.

One of the persistent myths about the Internet is that it is clean and green. We overestimate the value of going “paperless” while lowballing the effect on the environment of data centers. One need only look at an online pub like Data Center Knowledge to see that one of the most talked about issues in data centers today is how to reduce rack space, cooling and other energy costs associated with storage. (Another great resource is Greg Schulz’s StorageI/O blog.) This is particularly true of the data being generated through our new Web 2.0 sharing habits. Jon Toigo can laugh about the exploding digital universe all he likes, but it’s still the case that data growth is going like gangbusters in this socially networked era. Recession or no recession, there is a growing demand for ways to make storage more efficient.

Large players in this space are all too aware of the environmental and financial costs of such rapid data growth. Every time you share a photo or video, you’re contributing to it. And who among us doesn’t do this nowadays? In response. companies are experimenting with all kinds of techniques, including new building designs making use of outside air, reducing overall rack space usage with data reduction such as is offered by this blog’s parent Ocarina, cloud adoption, and so on and so forth. Companies like Google, Yahoo and Facebook are also creating next generation storage architectures that are more efficient for handling the realities of today’s internet. In short, let’s be sure, as we discuss the fallout from the latest global warming debate that we don’t start acting too lax about the effect of our actions on the planet.

What a long, strange trip it’s been for storage recently. First, there was the disaster that turned out to be less of a disaster than we thought. All those Paris Hilton phone numbers and Perez Hilton black book emails lost to a Sidekick in the ribs … But then it turned out all was not lost — in fact, most data was found, wagging their tails behind them.

Many in the industry have been howling about how unfair it was that the entire cloud took such a massive hit in light of the Sidekick meltdown. But then the skies cleared, and in his Enterprise Storage Strategies blog, Stephen Foskett came out with a surprising argument. All the more so as he works in the very industry that has been so massively hammered by the whole Danger/Sidekick/MS/EMC/HDS/cloud/whoever-else-we-can-blame disaster.

He writes: “Although my professional focus is at the forefront of the cloud storage wave, I can not disagree with the content of articles with sensational headlines like “Cloud Storage: It’s Strictly For Airheads” and “Why Cloud Storage Use Could Be Limited in Enterprises“. The authors are doing exactly what everyone should be doing: Questioning the viability and suitability of cloud storage in the enterprise.”

I agree, and would add that this should be the case when considering other solutions, including, yes, data reduction. In fact, our recent post on dedupe misconceptions has gotten all manner of attention recently for its even-handed response to alarmism about the necessity for dedupe for primary. It garnered a quick mention in Simon Sharwood’s roundup of storage blog-o-tweet-osphere smackdowns on SearchStorage. And there is plenty more to say about this–so much in fact that it doesn’t fit into this small space.

Not to worry, because over at his new Isilon blog, Nick Kirsch is asking for input on deduplication for primary, nearline and backups. Is it here to stay, or a craze? He asks. A good question, and one that might be answered with another question, “how fast is unstructured data growing?”

Finally, Online Storage Optimization hit the big time this week, getting a mention in the Forbes Velocity blog. Staffer Brian Caulfield, in search of a way to promote his new blog, called upon our wisdom of booth babes.

Happy weekend to you all.

Bo Peep Image: http://www.teachersandfamilies.com/nursery/bopeep.html

Friday’s CNBC segment is worth a watch if you’re wondering where the greatest growth will be over the next decade in the technology sector.

“Everything that we are using now–videos on YouTube. Everything we’re doing on our phone. On and on. It’s all about storage and putting stuff on the net,” opines Roger Nusbaum of Your Source Financial, who adds, “Cloud computing comes into play here.”

Not so funny story. My dad called me up today and told me that he received a letter from his credit card company, Chase. The letter stated that one of their tape backups had been lost. That tape contained information including his name, address, and social security number. The tape was held in a vendor’s warehouse, and had somehow been mislaid.

“You’re in this storage business. What do you think of this?” he asks.

“I think it shows that tape isn’t necessarily the safest way to back up data,” I answer. “Tapes can be mislaid, mishandled–even stolen. Let’s hope that latter possibility isn’t what happened here.”

According to the letter he received, the information on the tape is encrypted and would require “special software” to read. Also, the information that was lost didn’t include my father’s financial data, bank or credit card numbers. Still, he was worried by the letter. And I can understand that. For all the fears people have about the cloud, this is a fairly low tech example of how security breaches can occur. He said he didn’t know how many people were affected–or received the letter–but I’m waiting for a TechCrunch article on this to come out any day now.

Seems Uncle Sam is trying to tighten his belt through cloud computing. Government, at both the federal and state level, is debating how to make better use of this cost-cutting innovation. As we reported a few weeks ago, there are already some initiatives coming down the pike, such as Nirvanix providing cloud storage for NASA moon orbiter photos.

Now it looks as if NASA is getting seriously hooked on the cloud. As NextGov is reporting, the Obama administration is considering making NASA an IT service provider, using its cloud computing model in development, Nebula to manage and share all kinds of government data.

As the article explains:

“Federal CIO Vivek Kundra, Obama’s top technology executive, is examining many alternatives for innovation in the cloud, including using Nebula as a centralized platform to service multiple agencies, OMB officials said. Chris Kemp, CIO at NASA’s Ames Research Center, who is spearheading the program, is working with the federal government’s cloud working group, officials added.”

I know this sounds like a good idea in theory, but I do wonder whether it makes sense to trust the folks who brought us the Columbia should be entrusted with vast amounts of federal data. Just saying…

Vivek Kundra has also been busy coming up with federal cloud security standards. As Tim Greene reports in Network World, Kundra is proposing a “storefront model” in which a set of standards can “designate acceptable cloud service providers that government agencies can hire quickly without each agency having to independently determine that they are secure. The goal is to cut the cost and time needed to expand computing resources of government agencies by embracing the well known economic advantages of cloud computing.”

Meawhile, blogger Christopher Hoff (known to many as “Beaker”) says such a standard is already available, as he sketches out in a recent post on his Rational Survivability blog.

It’s not just the feds that are jumping on the cloud. (Which reminds me of this article from “The Onion,” but I digress.) Earlier this week, we read a report that a couple of Washington state legislators are attempting to derail a potential plan for a $300 million data center, because, they argue, the data it’s being built to store could be handled far more cheaply by a cloud provider.

Data Center Knowledge notes that they argue: “…Washington state is ‘home to many of the leading providers of this rapidly evolving commodity service … Still, our own state government has yet to move in this direction in any material way.’ Both Amazon (AMZN) and Microsoft (MSFT) are headquartered in the Seattle area and have in-state data centers that host cloud services.”

Some are concerned that this cloud mania might be getting out of hand. As one Twitterer, Bas Raayman argues–right now, all one has to say is “cloud” and people are lining up for it, without knowing whether it will really turn out to be a good match for the type of data they are trying to store and/or manage.

Time will tell whether this is a flash in the pan, or a flashy new plan that could save the government oodles of much needed dough.

A story caught my eye this morning that I just couldn’t help but blog about. Dave Simpson over at InfoStor is reporting that our friends at Nirvanix are providing cloud storage for a NASA mission to orbit the moon! As Simpson points out this is perhaps the most creative usage of cloud storage we’ve seen. I mean, really, how much more cool can you get with storage?

As Dave did, I will reproduce the entire press release here:

Nirvanix Brings Storage from the Moon to the Cloud with Successful Launch of Lunar Reconnaissance Orbiter

Comprehensive imagery data from onboard cameras providing deeper understanding of the moon and its environment to be copied to CloudNAS-based solution

San Diego, June 29, 2009 – An Atlas V 401 rocket carrying two lunar satellites launched from Cape Canaveral Air Force Station in Florida at 5:32 p.m. EDT on June 18th in what is being described as America’s first step to the lasting return to the moon. One of the satellites, the Lunar Reconnaissance Orbiter (LRO), will begin to provide high-definition imagery of the moon once in orbit with a copy of all data stored on the Nirvanix Storage Delivery Network™ via CloudNAS®, a software based gateway to secure enterprise cloud storage.

After a four-day trip, the LRO will begin orbiting the moon, spending at least a year in a low polar orbit collecting detailed information about the lunar environment that will help in future robotic and human missions to the moon. Images from the Lunar Reconnaissance Orbiter Camera will be transmitted from the satellite to a project team at Arizona State University for systematic processing, replicated to secondary high-performance storage in a separate building at ASU and then replicated to the Nirvanix Storage Delivery Network (SDN™). Nirvanix provides a method for storing a tertiary copy of the data offsite by installing CloudNAS and writing a copy directly from the data-receiving servers. ASU and NASA have already transferred multiple TBs of original Apollo mission imagery to the Nirvanix CloudNAS-based solution.

“While this project may be one small step for NASA’s program to extend human presence in the solar system, it definitely represents a giant leap in cloud storage’s ability to provide a reliable, scalable and accessible alternative to tape for long-term retention of enterprise-class data,” said Jim Zierick, President and CEO of Nirvanix. “The tertiary copy of images from the LRO Camera stored on the Nirvanix CloudNAS is online and accessible within seconds and the project managers at ASU do not need to worry about managing offsite storage, allowing them to focus on the more important mission at hand. We are pleased to be part of such a historic project and value our contribution to finding a deeper understanding of the moon and its environment.”

Nirvanix CloudNAS is a fast, secure and easy way to gain access to the benefits of Cloud Storage. As the world’s first software-only NAS solution accessible via CIFS or NFS, CloudNAS offers enhanced secure data transfers to any of Nirvanix’s globally distributed storage nodes using integrated AES 256-bit encryption and SSL options. Through the Nirvanix CloudNAS, organizations have access to unlimited storage via the Nirvanix Storage Delivery Network with the ability to turn any server on their network into a gateway to the cloud accessible by many existing applications and processes.

About Nirvanix

Nirvanix is the market leader in enterprise cloud storage and offers a fully managed cloud storage service designed from the ground up for the enterprise. Headquartered in San Diego, California, Nirvanix is privately held and has raised more than $23 million in funding from world-class investors including Intel Capital, Valhalla Partners, Mission Ventures, Windward Ventures, and the European Founders Fund. Nirvanix has over 500 customers ranging from Internet startups to Fortune 10 organizations and include Arizona State University, Nero, and The Planet, among others.

For more information about the company and its services, visit www.nirvanix.com.

Note: this is the first in a series of posts on the blogs that make up the Online Storage Optimization blogroll. Please look out for future reviews of other storage bloggers.

Every once in awhile I find myself enjoying a blog so much that I end up reading several posts in one sitting. Such was the case today with Storagebod’s Blog. Who else, I thought, could integrate references to Winnie-the-Pooh with cloud storage while making subtle points about storage infrastructure costs? This must be a sign I’m becoming a fan.

Storagebod, whose real name is Martin Glassborow, is an independent storage blogger whose topics cover a wide swath of storage and tech-related topics. His bio states that he’s responsible for storage infrastructure for a large UK Media company, which he doesn’t name. He also says in posts that he utilizes both EMC and NetApp storage, which puts him in an interesting position vis a vis the two competitors.

I’ve gotten chatting with Martin on Twitter on several occasions (as have some other contributors to this blog), and one thing that stands out about him is that while he has strong opinions about storage products, they always seem to come from a customer perspective — that is, he’s not interested in slamming a vendor for its own sake. Rather, he takes a pragmatic approach that speaks to a larger mission of helping other storage and IT professionals who are also struggling to control costs, keep data safe, and so on.

So, even while mocking IBM’s latest cloud offerings with his Milne-inspired ditty, he gives it the benefit of the doubt, saying, “…I’ve been a bit unfair, it’s not just tin, it comes with a raft of management software as well…”

Another recent post about a recent Amazon AWS outage doesn’t slam the company for losing a data center, but instead argues for better planning for such an eventuality.

“When Amazon lose a data-centre in their cloud, this should not be news! It will happen, it may be a whole data centre, it may be a partial loss. This not a failure of the Cloud as a concept; it is not even a failure of the public Cloud…”

In short, this is a blogger I recommend for anyone who would like to read spirited, opinionated yet fair coverage of storage from the point of view of someone who knows your pain. And while he never seems to quite find the best way to alleviate it, the process he goes through should be enlightening to many, both within and outside the industry.

If Tom Lehrer could have an album called “Revisited” then there’s no reason why this blog can’t do the same thing with a piece by Carter George on cloud storage.

When we ran this post last May, cloud storage was an important topic of discussion. Since then, the blogo-tweet-o-sphere has exploded, with numerous posts per week on the subject, such as this one and this one.

Here’s an excerpt of what Carter had to say a little under a year ago about cloud storage. Still rings true, I’d say:

Most customers will still look at cost/Gigabyte as the main motivator to use a service like this and, at the right price and benefit point, people will put their files online. Since all these storage service providers all buy their disks from the same small number of companies that actually make disk drives, the costs are all roughly the same for the physical infrastructure needed to build an online storage service and compete.

I think that the real solution here is that, for anyone to breakthrough and get some separation from the crowd, they are going to have to incorporate breakthrough storage optimization in their offering – and do so in a way that’s transparent to the end user. That could be dedupe, that could be compression, or it could be something more sophisticated like Ocarina. The main thing is that if you can get 5:1 or 10:1 ratios on how much logical space you can provide via the cloud to how much physical space you, as a provider, have to buy, then you can have a compelling proposition. The competition is fierce in this market and in order to grow and thrive in any business that offers online storage, the providers are going to have to develop a strategy to significantly increase their online storage capacity without increasing cost and overhead in step.

 Optimization includes both compression and content-aware dedupe, and effects both how much it costs to store files in the cloud, as well as optimizations that would make uploading to, and reading from, cloud storage faster over the internet.        

Because almost all clouds are based on “forests” of industry standard servers with software to tie them together as a self-healing scalable storage pool, they have the ideal architecture for hosting lots of CPU-intense data reduction algorithms – next-gen object dedupe, and content-aware compressors that work on specific file types. A traditional filer does not have that kind of CPU horsepower – so the cloud is not only a different cheaper place to go rent Terabytes out. It’s also a new green-field architecture on the storage side – whether you are talking Mozy, Nirvanix, Zetta, or Microsoft – with the kind of horsepower to host new fundamental features for cost- and capacity-optimized storage.

No one can miss the fact that data storage is spiraling upward at a terrifying rate. Joerg Hallbauer puts it on Dell’s Future of Storage blog hit the nail on the head with his post: “We are running out of places to put things.”

Citing data collected by IDC, Hallbauer concludes that in a mere three years, we there will be 1400 exabytes sitting on disk. Currently, according to the study, there are 281 exabytes of data being stored, and the CAGR rate is 70 percent. Much of this data is on laptops, home computers or servers under your desk today, but as Joerg correctly notes, there’s no question its migrating quickly to the cloud. Huge data centers will end up holding most of this data, and disk drives are not growing fast enough to deal with it anymore.

So, where do we go from here? Well, if the traditional answer was, wait for bigger drives so I can put more stuff on a disk, the other logical thing to do is to say, how can I put a lot more stuff on disks that I already have?  The answer is advanced storage optimization. The first simple storage optimization solutions are out there today – single instancing, deduplication, and compression. But the area of storage optimization is really just taking off, and much more sophisticated approaches are emerging that will allow a disk – whatever its physical size – to store 10, 20, or 100 times more data than it does today.  

What’s more, the move to large data centers providing huge cloud storage services will make this more efficient, because storage optimization is all about finding redundant information and figuring out how to store it more efficiently. So the larger the data set, the more likely you will see big wins from next generation storage optimization.

This also naturally leads to more tiering. Where today you have fast disks (Fibre Channel or SAS) and slow disks (SATA) making up the tiers, it’s much more likely in the future that the fast tiers will be solid state storage of some sort (SSD and Flash, as Joerg points out) and the massive tiers that hold the bulk of all these Exabytes will be the largest possible disks integrated in to systems that have very efficient storage optimization built in.

Image credit: Orange Photography blog archives