Online Storage Optimization

With all the talk about the data inconsistencies around climate change theory, one issue that I’d hate to see lost in the shuffle is the actual environment. That is, while I personally have been skeptical for some time about the alarmist tone many scientists took regarding global warming, it would be a shame if there was such a backlash that people forget about the much more crucial, larger issue at stake. That is, we need to look at all the ways –on macro- and micro-scales–that we can reduce the overall pollution we generate through our daily habits.

One of the persistent myths about the Internet is that it is clean and green. We overestimate the value of going “paperless” while lowballing the effect on the environment of data centers. One need only look at an online pub like Data Center Knowledge to see that one of the most talked about issues in data centers today is how to reduce rack space, cooling and other energy costs associated with storage. (Another great resource is Greg Schulz’s StorageI/O blog.) This is particularly true of the data being generated through our new Web 2.0 sharing habits. Jon Toigo can laugh about the exploding digital universe all he likes, but it’s still the case that data growth is going like gangbusters in this socially networked era. Recession or no recession, there is a growing demand for ways to make storage more efficient.

Large players in this space are all too aware of the environmental and financial costs of such rapid data growth. Every time you share a photo or video, you’re contributing to it. And who among us doesn’t do this nowadays? In response. companies are experimenting with all kinds of techniques, including new building designs making use of outside air, reducing overall rack space usage with data reduction such as is offered by this blog’s parent Ocarina, cloud adoption, and so on and so forth. Companies like Google, Yahoo and Facebook are also creating next generation storage architectures that are more efficient for handling the realities of today’s internet. In short, let’s be sure, as we discuss the fallout from the latest global warming debate that we don’t start acting too lax about the effect of our actions on the planet.

It’s been a tough year — a worldwide recession, a sluggish housing market, rising unemployment … and on top of all that, the tarnished image of one of sports’ most squeaky clean players. Well, actually, there have been some bright spots. As DCIG blogger and storage analyst Jerome Wendt notes while looking back at the past year, “Deduplication is the Big Success Story of 2009.”

Wendt writes: “Deduplication is arguably one of the most notable trends of 2009 as it has been widely adopted by users after bursting onto the scene just a few years ago and has grown to be included in both software and hardware products.”

Wendt focuses on dedupe for backups, where there has been much publicized activity over the past year. The big storage story of 2009 was of course the battle between storage titans EMC and NetApp over backup dedupe specialist Data Domain. He cites an industry survey from SearchDataBackup that indicates that 41% of enterprises either are or are seriously considering dedupe to control data growth and costs. He also notes that the despite the predicted demise of Quantum, that dedupe company remains strong.

Dedupe for backups is one part of the cost reduction puzzle. Another part is to reduce data at the source, in primary storage. This is of course the specialty of this blog’s parent Ocarina, which implements a unique combination of content-aware dedupe and compression to achieve startling results. It focuses on the very types of unstructured data that are driving storage growth today–emails, images, documents, and so on. The company has been partnering with almost every leading storage provider, including HP, EMC, HDS, BlueArc, and Isilon. Another  leader in this space is NetApp, which has a strong dedupe for primary offering that has also garnered a great deal of attention.

Here’s the thing, the economy might be slowing down, but data growth continues apace. This is one reason that the storage industry has been thriving this year. But rather than standing still, what is spells is a concerted effort to keep that data under control. As Wendt notes, another of the year’s big trends is cloud storage, which offers companies more flexibility for storing some percentage of their data. I would also add that virtualization has taken a huge leap forward, not only in terms of the technology itself, but also in terms of adoption over the past year. Yet another way to attack the problem.

So if 2009 was all about dedupe for backups, I’m going to guess that 2010 will be very much about data reduction at all points on the data life cycle. What do you predict?

Image: Gizmodo

Two items jumped out at me this week. First, Francine Hardaway’s post on being an early adopter, “Do You Write in the Cloud?” and second, this New York Times story about Cormac McCarthy’s typewriter.

Hardaway, a serial entrepreneur and founder of Stealthmode Partners talks about how she and other bloggers compose their text. Mashable’s Pete Cashmore, along with Buzzmachine’s Jeff Jarvis and Gina Trapani all write in the cloud. Hardaway either writes directly in Gmail or sends one to Posterous from her Gmail account. If there is a bad wifi connection, she uses Evernote and synchs it to the cloud before moving it to Wordpress. (Am I the only one who composes directly into Wordpress?)

The main point here is that Hardaway illustrates how she was ahead of the curve each time, and was able to show successfully that you could still think and write using the latest “newfangled” tech–whether a manual typewriter or a word processor. Hardaway recollects:

“…50 years ago, I started composing on a manual typewriter when most other people composed in longhand and transcribed.  Many famous authors wrote in longhand their entire lives. People thought I was crazy for going straight to the typewriter. But I’ve gone seamlessly from the typewriter to the word processor (we could dictate into a  Digital Equipment Corporation word processor in the Maricopa Community Colleges in the 70s, when I was a professor.)  They thought I was crazy for dictating instead of typing when I did that.”

All of this was sloshing around in my brain when I came upon the aforementioned story of Cormac McCarthy’s typewriter. McCarthy is the author of a number of bestselling books, including the recent “The Road“–now a movie, and “All the Pretty Horses,” among many other award winners. How did he compose his text? On a portable Olivetti typewriter he bought at a pawn shop in 1963. He moved on from long hand, but after that he just stopped.

We’re talking about two ends of the spectrum. The early early adopter. And the … well, I don’t think we can call Cormac McCarthy a late adopter now can we? Of course, most people are somewhere in the middle. Enterprises are managing an influx of data that is based on what would be considered the median behavior–a need to store all manner of files, from Office documents to family snapshots to the latest Lady Gaga download. Oftentimes, the response is to try to ignore the problem. More and more files are shoved onto expensive primary storage. Suddenly, the problem reaches massive proportions and emergency measures are taken, rather than a well thought out tiering/deduplication plan or a move to a reliable cloud storage platform. But I digress. The main question here is, what kind of adopter are you? And what are the strengths and weaknesses of each approach?

What a long, strange trip it’s been for storage recently. First, there was the disaster that turned out to be less of a disaster than we thought. All those Paris Hilton phone numbers and Perez Hilton black book emails lost to a Sidekick in the ribs … But then it turned out all was not lost — in fact, most data was found, wagging their tails behind them.

Many in the industry have been howling about how unfair it was that the entire cloud took such a massive hit in light of the Sidekick meltdown. But then the skies cleared, and in his Enterprise Storage Strategies blog, Stephen Foskett came out with a surprising argument. All the more so as he works in the very industry that has been so massively hammered by the whole Danger/Sidekick/MS/EMC/HDS/cloud/whoever-else-we-can-blame disaster.

He writes: “Although my professional focus is at the forefront of the cloud storage wave, I can not disagree with the content of articles with sensational headlines like “Cloud Storage: It’s Strictly For Airheads” and “Why Cloud Storage Use Could Be Limited in Enterprises“. The authors are doing exactly what everyone should be doing: Questioning the viability and suitability of cloud storage in the enterprise.”

I agree, and would add that this should be the case when considering other solutions, including, yes, data reduction. In fact, our recent post on dedupe misconceptions has gotten all manner of attention recently for its even-handed response to alarmism about the necessity for dedupe for primary. It garnered a quick mention in Simon Sharwood’s roundup of storage blog-o-tweet-osphere smackdowns on SearchStorage. And there is plenty more to say about this–so much in fact that it doesn’t fit into this small space.

Not to worry, because over at his new Isilon blog, Nick Kirsch is asking for input on deduplication for primary, nearline and backups. Is it here to stay, or a craze? He asks. A good question, and one that might be answered with another question, “how fast is unstructured data growing?”

Finally, Online Storage Optimization hit the big time this week, getting a mention in the Forbes Velocity blog. Staffer Brian Caulfield, in search of a way to promote his new blog, called upon our wisdom of booth babes.

Happy weekend to you all.

Bo Peep Image: http://www.teachersandfamilies.com/nursery/bopeep.html

This year’s VMWorld ended with a bang–as rock legend Foreigner played for a crowd of 10,000 virtualization fanatics at the Yerba Buena Center in S.F. This really was one heck of a jampacked and exciting event. It is clear that this is becoming one of the key shows for the virtualization and storage markets. There were two–count ‘em!–EMC acquisitions. The big buzzword was “cloud.” There were more than a few product launches, and even a few new companies on the scene.

On a more personal note, this was a great way to connect with a whole lot of storage bloggers that I’d met online but not in person. The storage “tweetup” on Tuesday evening drew a crowd that could be called a veritable who’s who of the storage blogging community, including Marc Farley, Stephen Foskett, “HPStorageGuy” Calvin Zito, NetApp’s Alex McDonald, Dave Vellante of Wikibon, Ed Saipetch, “Storagenerve” Devang Panchigar, and others. I also met EMC bloggers Dave Graham and Chad Sakac, and NetApp blogger Vaughn Stewart. What a talented community!

One thing that made VMWorld stand out was the diversity of companies represented there–from huge heavyweights such as EMC to newish tech cos. like Veeam to startups just coming out of stealth. Perhaps the hottest of these new entrants was EvoStor–which I mentioned in yesterday’s post. As Storage Switzerland’s George Crump says in a recent VMWorld dispatch on Byte and Switch:

“EvoStor, a VMware only, first that I have seen, storage solution. This is more than focus, this is exclusive support of a single environment … The system is basically a storage grid similar to other scale-out models like LeftHand or Isilon but again, focused on VMware.”

Also this week, EMC went on a shopping spree. Tuesday, we learned it had picked up e-discovery specialist Kazeon. As Stephen Foskett notes in Gestalt IT, “EMC will likely integrate the Kazeon technology with their SourceOne archiving and discovery platform.” He also says that while EMC probably paid a fair price for the technology–just $75 million–it’s not without its challenges.

A day earlier, on the first day of VMWorld, EMC announced it had acquired “application image management” company FastScale. On his blog, Chuck Hollis describes this as the final piece in a puzzle that it has been assembling for some time with Ionix. He calls their method of prepping images for virtualization “predupe,” reducing footprint in memory.

As Beth Pariseau writes in Storage Soup, the FastScale acqui is a sign that EMC and VMWare are acting more like two interrelated companies. The more logical acquirer would be VMWare, since this is a server virtualization company. This could signal a shift in focus away from their purely hands-off approach that meant that it sometimes even signed agreements with EMC arch-competitor NetApp. Time will tell where this is leading.

There’s much more to say about the show, but this will have to do for now.

Not so funny story. My dad called me up today and told me that he received a letter from his credit card company, Chase. The letter stated that one of their tape backups had been lost. That tape contained information including his name, address, and social security number. The tape was held in a vendor’s warehouse, and had somehow been mislaid.

“You’re in this storage business. What do you think of this?” he asks.

“I think it shows that tape isn’t necessarily the safest way to back up data,” I answer. “Tapes can be mislaid, mishandled–even stolen. Let’s hope that latter possibility isn’t what happened here.”

According to the letter he received, the information on the tape is encrypted and would require “special software” to read. Also, the information that was lost didn’t include my father’s financial data, bank or credit card numbers. Still, he was worried by the letter. And I can understand that. For all the fears people have about the cloud, this is a fairly low tech example of how security breaches can occur. He said he didn’t know how many people were affected–or received the letter–but I’m waiting for a TechCrunch article on this to come out any day now.

George Crump has a piece in Byte and Switch today that poses an important question: “Can we get to a single point of deduplication?” This is a question that we have taken up in one form or another in some of our recent posts, such as this one and this one.

In the article, Crump asks the question in another way: “… can you have all your data tiers; primary, archive and backup deduplicated by a single engine?”

In light of the recent focus on deduplication, this in my view is a question that really does need to be raised. For how long will the industry to silo out these different tiers for its deduplication solutions? And how much sense does it make to rehydrate data every time you move it, in order to once again deduplicate it? Not a lot.

Crump writes: “The current deduplication vendors could work on building out their solutions to either scale up into primary storage performance (see Data Domain’s DD880) or they could move their existing data duplication technology into other markets; see the increased speed of Ocarina Networks and Permabit as well as their move into cloud storage.”

At the same time, as we’ve pointed out here, online storage is quite a bit different than backups and so far at least, none of the successful backup dedupe vendors - Data Domain, Diligent, Quantum, etc. have been able to break into it. Rather, it is NetApp and Ocarina who have been the trailblazers.

Crump makes another key point:

“NetApp and Ocarina could continue to enhance and improve the re-hydration speed of their technologies to make read performance a non-issue, making primary storage a viable platform. Ocarina can already maintain the deduplicated format as they move through tiers, so landing on backup or archive disk would simply be another move for them.”

This is an interesting observation, and one that is often missed in reporting on both of these solutions. We look forward to seeing more debate and discussion on this issue, which was well kicked off with this piece.

Seems Uncle Sam is trying to tighten his belt through cloud computing. Government, at both the federal and state level, is debating how to make better use of this cost-cutting innovation. As we reported a few weeks ago, there are already some initiatives coming down the pike, such as Nirvanix providing cloud storage for NASA moon orbiter photos.

Now it looks as if NASA is getting seriously hooked on the cloud. As NextGov is reporting, the Obama administration is considering making NASA an IT service provider, using its cloud computing model in development, Nebula to manage and share all kinds of government data.

As the article explains:

“Federal CIO Vivek Kundra, Obama’s top technology executive, is examining many alternatives for innovation in the cloud, including using Nebula as a centralized platform to service multiple agencies, OMB officials said. Chris Kemp, CIO at NASA’s Ames Research Center, who is spearheading the program, is working with the federal government’s cloud working group, officials added.”

I know this sounds like a good idea in theory, but I do wonder whether it makes sense to trust the folks who brought us the Columbia should be entrusted with vast amounts of federal data. Just saying…

Vivek Kundra has also been busy coming up with federal cloud security standards. As Tim Greene reports in Network World, Kundra is proposing a “storefront model” in which a set of standards can “designate acceptable cloud service providers that government agencies can hire quickly without each agency having to independently determine that they are secure. The goal is to cut the cost and time needed to expand computing resources of government agencies by embracing the well known economic advantages of cloud computing.”

Meawhile, blogger Christopher Hoff (known to many as “Beaker”) says such a standard is already available, as he sketches out in a recent post on his Rational Survivability blog.

It’s not just the feds that are jumping on the cloud. (Which reminds me of this article from “The Onion,” but I digress.) Earlier this week, we read a report that a couple of Washington state legislators are attempting to derail a potential plan for a $300 million data center, because, they argue, the data it’s being built to store could be handled far more cheaply by a cloud provider.

Data Center Knowledge notes that they argue: “…Washington state is ‘home to many of the leading providers of this rapidly evolving commodity service … Still, our own state government has yet to move in this direction in any material way.’ Both Amazon (AMZN) and Microsoft (MSFT) are headquartered in the Seattle area and have in-state data centers that host cloud services.”

Some are concerned that this cloud mania might be getting out of hand. As one Twitterer, Bas Raayman argues–right now, all one has to say is “cloud” and people are lining up for it, without knowing whether it will really turn out to be a good match for the type of data they are trying to store and/or manage.

Time will tell whether this is a flash in the pan, or a flashy new plan that could save the government oodles of much needed dough.

Great line-up of speakers at Storage Networking World next week in Orlando. Among other highlights, Carter George, VP Products at Ocarina Networks and the lead author here on Online Storage Optimization will give a talk entitled “Building Castles in the Cloud:  Leveraging Cloud Storage Architecture to Lower Costs” on Wednesday, April 8 at 3:05 p.m. Anyone who is interested in seeing how cloud storage can solve multiple business problems should attend. Because of Ocarina’s role in the storage marketplace, Carter has a unique perspective on how cloud storage architectures have evolved, and how they are in a position to influence the direction of the entire storage industry.

Here’s the talk in a nutshell:

Cloud storage is one of the fastest-growing segments in the storage industry due to its potential for lowering costs and moving the burden of managing and protecting data to the cloud provider. It has given rise to new innovative storage architectures, and could well be a harbinger of future corporate storage architecture. In this talk, we’ll look at cloud storage architecture across multiple cloud providers, offering a detailed look at the technology trends and innovations in cloud storage platforms, including the use of grid architectures of commodity storage nodes and the use of global namespaces. We will also discuss how cloud grid architectures enable value-add storage software breakthroughs, such as integrated content-aware compression, dedupe, replication instead of backup, replication by file type policies, and the move of value in storage to the software layer, rather than in legacy vendor hardware. Vendor offerings to be discussed include: Amazon, IBM, Nirvanix, and others.

We hope to see you at SNW.

More cloud storage news today, as ParaScale is offering software that allows companies to turn a network of commodity Linux servers into a cloud storage platform. Sony Pictures Imageworks and Stanford Genome Technology Center are testing it with an eye towards using it to manage their massive image rich storage needs.

Good coverage of the announcement by Lucas Mearian at Computerworld and Dave Raffo at SearchStorage.

In the press release, Noemi Greyzdorf, storage analyst at IDC is quoted saying “‘ParaScale’s flexible, object-oriented file system allowing scale and ease of management across different pools of storage and will enable content and general tier2 storage deployments to achieve new levels of economy and flexibility.’”

On a related note, Chris Evans has written a thoughtful post on cloud storage and availability questions on his Storage Architect blog.

Looking forward to hearing the speculation across the blogosphere about what if anything this vendor is offering that’s new.