Online Storage Optimization

Four out of five college students agree, this is not the way to deal with data growth. How about this instead?

I’ve noticed a few blog posts recently about speed of deduplication in the modern data center. I agree that speed is an important factor, but keep in mind that not all dedupe is created equal. That is to say, fast is good, but only if you are also effective. One of the tricky things has been that the easiest data to compress is also usually the most carefully performance tuned. A great example of this is a database. This is because databases are comprised of simple alphanumeric fields and sparse tables. All of that is easy to reduce in size.

However, a company’s core transactional database is the most conservative asset in the data center. Introducing compression would save space, for sure, but you could only use very fast, simple compressors there. At the same time, customers will be hesitant to deploy a new layer of processing in their most sensitive application.

So, where is most data growth? In fact, it’s being driven by unstructured data – Office documents, rich media, email with attachments, PDFs, Flash videos, and so forth. This complex data does not lend itself to fast simple compressors. But perhaps we should back up for a moment and think about how customers have been behaving all along.

Throughout the history of storage, there have always been tradeoffs available between fast expensive storage, and slower but cheaper alternatives. This is not a bad thing. It gives users alternatives based on their priorities and budgets. Back in the old mainframe days, these choices were between very expensive mainframe memory and “offline” storage like drums, cards, and tapes. Today the technology is all much bigger, faster, cheaper and sexier. But really, the tradeoffs are the same.

Data reduction technology adds another layer of choice above and beyond the traditional hardware choices. Now in addition to choosing whether you want fast, expensive solid state disk (SSD) or slower but very cost-effective SATA, you can also choose whether you want to compress and/or deduplicate the data that is stored on those disks.

Just like physical disks, compression and dedupe come in a range of speeds and capabilities. There are simple and very fast compressors that are essentially invisible in terms of their impact on storage performance. There are more complex compressors that get better results, but which may take longer, either to compress or to decompress the data. Deduplication, done well, should always be pretty fast, and streaming dedupe rates of well of 300MB/sec are now available from many vendors (including Data Domain and Ocarina).

The emergence of tools to automatically tier data to its appropriate place help make the use of all of these technologies more feasible. That applies as much to solid state disks as it does to dedupe and compression. When data tiering can be made invisible to end-users and applications, then implementing multiple physical and logical tiers of storage becomes practical.  Good examples would include EMC’s new FAST tools, Compellent’s “Fluid Data Storage”, and HDS’s Data Migrator. When users or administrators have to move data by hand to get it to a compressed tier or a solid state disk, then the operational costs offset the capital savings.

You might want to be wary when someone’s biggest claim to fame is fast dedupe. Just as the old mainframe admin had to decide whether something was important enough to live in RAM, or could be stored on cheaper tapes instead, today’s IT shops have to decide where it is most important to try to get data reduction, and what tool will get the most bang for the buck for that kind of data. You need the whole story, and then you can decide based on your own priorities.

Interesting story from the vault of the Ocarina case study library. Social network Tagged is the third largest social network in the U.S. It has seen traffic increase 10x over the past two years. With its focus on making new friends rather than simply getting to know existing ones, it has carved out a successful niche and is building an international subscriber base of over 80 million members.

The cost of this success? Data growth. Tagged’s storage infrastructure has been doubling every single year. With 1 million new photos uploaded every single day, Tagged needed a way to expand capacity and fast.

Compression with Ocarina meant about 10 TB of additional free space, which in turn meant they could put off buying new NAS equipment by several months. The lower average image size also meant reduced bandwidth and 15%-20% reduced monthly content delivery network (CDN) costs.

The company chose to go with Ocarina’s newest specialized image reduction technique, native format optimization (NFO). This is visually lossless compression of images that nevertheless delivers significant space savings–a technology that’s perfectly suited to the social networking environment.

The other crucial benefit to reducing image size was improvements in site responsiveness. “We’re sure that using Ocarina to reduce image sizes has helped improve our page rendering times,” said company CTO Johann Schleier -Smith. “That’s a big deal because it creates a better user experience, which means improved customer loyalty and higher market share.”

Read the entire case study by clicking here. Or visit the Ocarina resources page and click on the Case Studies tab, where you’ll find several others.

This past year, we at Online Storage Op gathered all manner of images to illustrate our posts. So as a way of looking back at 2009, here are some of the ones we liked the best–and the stories that went with them:

Holodeck fun:

In February, Robin Harris at StorageMojo wrote about a potential breakthrough in storage technology that could change the landscape forever: quantum holographic storage. Online Storage Op was on the scene. It also gave us a chance to upload a pic of a Geordi La Forge doll. Admit it… this is one cool toy.

Squeezing into your Genes:

This blog’s parent Ocarina had quite a year–inking partnerships with a number of major storage vendors and becoming a noted player in the hot dedupe space. It was also the year that genomics labs woke up to the need for better data reduction to deal with the coming onslaught of genetic data. In short, compression can be a matter of life and death. We reported on it here, and our readers got to relive their 10th grade biology class by looking at images like the one above.

Racing for Dedupe

As many pundits are now opining, dedupe really was one of the biggest stories of 2009, not least because of the high profile battle for Data Domain between storage titans EMC and NetApp. In the end, EMC nabbed the dedupe specialist for an eye-popping $2.1 billion.

Booth Babe Mania:

We know our readers are sophisticated types who come here only to absorb information and opinion, and to better themselves for the benefit of all humankind. But for some odd reason we saw a major traffic spike the day we ran our post on the great Booth Babe Controversy. When we asked, everyone quickly told us, “I read the articles.” Mmmhmm!

VMworld a hit

And speaking of images that make storage folks drool, one of the most mesmerizing sights of the year was at VMworld, held in August in San Francisco. Participants descended the escalator to be greeted by gleaming rack of servers and storage–which we later learned was the result of a plan drawn on a napkin by the VMware GETO team. In any case, this year’s VMworld was a major event–and as we rightly noted, it foretold more economic activity in storage and virtualization.

Industry puts aside differences to try to save a life

This is one of the saddest stories of 2009, and one that demonstrates an activist and caring streak in the storage community. When word got out in May 2009 that EMC employee Nick Glasgow was in need of a bone marrow transplant, folks within the storage industry put aside competitive differences and pulled together to find him a match. Sadly, Nick passed away in October. The degree to which he inspired others will not be forgotten.

And, finally…

We never did have an egg and spoon race, but…
In November, Ocarina participated in the first ever Gestalt IT Tech Field Day, which brought independent bloggers from around the world to Silicon Valley for two days of tech deep dives. Our “bring out your data” challenge started tongues wagging well before the event began. Participants brought us their toughest data sets, and aside from those who used archaic encryption software to stump our algorithms, the results were impressive–an average of about 30% reduction on these tougher-than-tough data sets. Plus, the whole event was just a ton of fun. And it didn’t even require that we slog around the mud clapping coconut shells together.

Tis the week for the “out of office” email messages. But the storage blogo-tweet-osphere waits for no man. Here are a few posts that caught my eye this week.

Bas Raayman sees CPU power hitting the wall: The RAM per CPU wall

Rick Vanover says 2010 could be the year for 10GigE - Will 2010 see 10 Gigabit Ethernet go mainstream?

It being the end of a year–and a decade–predictions abounded. We’re pleased to note that when it came to summarizing the top storage stories of 2009, deduplication for primary storage, the specialty of this blog’s parent Ocarina, made the big lists:

Infostor: The top 5 storage technologies of 2009 (and 2010?)

“Storage optimization (or data reduction) technologies such as data deduplication and compression can significantly reduce capacity requirements and costs … Consider data reduction for primary storage.”

SearchStorage - Beth Pariseau: Top 10 enterprise data storage news stories of 2009

“10. Data deduplication branches out. As deduplication settled into a comfortable role in backup, data-reduction technology started working its way into other parts of the data storage infrastructure, including primary as well as nearline and archived data … Ocarina and Isilon Clustered NAS help visual effects studio archive images, cut costs.”

For sheer inventiveness, blogger Stephen Foskett wins the prize with his 2009 predictions post, in which he turns the clock back and takes advantage of 20-20 hindsight: My 2009 IT Industry Predictions.

Meanwhile, social media and tech watcher Louis Gray takes himself to task and looks at all of his 2009 predictions to see how well he fared: My 2009 Tech Predictions: Mixed, But Nailed Real-Time.

OK that’s all for now. Here’s wishing all of you a happy, healthy, green and techy new decade.

George Crump of Storage Switzerland has a post up today on Network Computing that really should be made required reading for every single mid-level executive in the storage industry and beyond. He offers the following simple, straightforward advice:

“…every vendor’s opening slide should have three bullets: The exact problem the solve, why they are different, and how much the product costs. Optionally, the slide could end with a ‘Should we continue? Yes or No?’”

Leaving aside the wisdom of putting a price up on the very first slide, the main point–what problem do you solve?–is the crucial piece here. When you step back and look at it, this is one of those more than obvious pieces of advice. Why else are you giving the presentation? Yet, I dare you to take a look at your slides–the ones you present to analysts, prospects, and others. What does the first one say? I thought so.

If you’re not fixing something that’s broken, why are you in business? Or, if you don’t like the negativity, you can use a phrase that my friend Tony Asaro at Contemplating IT prefers, you’re addressing “pent up demand.” That’s fine too–in fact it might be a more accurate way of expressing the situation.

The main thing is that you’re addressing some THING, not doing a creative but ultimately useless science research project. And most likely you are. But many companies are so enamored with their cool tech they forget all about the purpose of their existence. Their slides (and web sites, and press releases and other promotional platforms) are all focused inward on themselves, rather than on their raison d’etre–their customers.

Here’s an example of a problem that needs solving. Data growth is raging out of control. As it happens, it is unstructured data that is posing the biggest threat to data centers worldwide–email, documents, photos, videos, sound files–in short, the detritus of our wired age. The problem isn’t getting better–in fact, it’s worsening with each passing day. Despite the fact that the cost of disk is going down, the environmental and monetary price to be paid for cooling, power and rack space could cripple a number of otherwise healthy concerns.

Here’s how Graham Hobson, CTO of Photobox, Europe’s largest photo sharing site, explained his problem to me when I interviewed him last spring: “Our data was growing by leaps and bounds–on busy days, millions of photos are uploaded to our servers. If we were to fill a typical data center rack with storage systems, we needed 32 amps of power. But a lot of these data centers in Europe are really only geared up to provide 8 or 16 amps per rack. They were designed for telecom. So there are very few suppliers we can go to that have rack space and that power quota. And of the people that remain they’re not really inclined to give discounts, because their costs are rising.” In short, the company was literally running out of space to store their customers’ data.

That constitutes a serious and pressing issue in my book. What problem does your company solve? Is it spelled out on the first slide of all of your presentations? If not, why not?

It’s been a tough year — a worldwide recession, a sluggish housing market, rising unemployment … and on top of all that, the tarnished image of one of sports’ most squeaky clean players. Well, actually, there have been some bright spots. As DCIG blogger and storage analyst Jerome Wendt notes while looking back at the past year, “Deduplication is the Big Success Story of 2009.”

Wendt writes: “Deduplication is arguably one of the most notable trends of 2009 as it has been widely adopted by users after bursting onto the scene just a few years ago and has grown to be included in both software and hardware products.”

Wendt focuses on dedupe for backups, where there has been much publicized activity over the past year. The big storage story of 2009 was of course the battle between storage titans EMC and NetApp over backup dedupe specialist Data Domain. He cites an industry survey from SearchDataBackup that indicates that 41% of enterprises either are or are seriously considering dedupe to control data growth and costs. He also notes that the despite the predicted demise of Quantum, that dedupe company remains strong.

Dedupe for backups is one part of the cost reduction puzzle. Another part is to reduce data at the source, in primary storage. This is of course the specialty of this blog’s parent Ocarina, which implements a unique combination of content-aware dedupe and compression to achieve startling results. It focuses on the very types of unstructured data that are driving storage growth today–emails, images, documents, and so on. The company has been partnering with almost every leading storage provider, including HP, EMC, HDS, BlueArc, and Isilon. Another  leader in this space is NetApp, which has a strong dedupe for primary offering that has also garnered a great deal of attention.

Here’s the thing, the economy might be slowing down, but data growth continues apace. This is one reason that the storage industry has been thriving this year. But rather than standing still, what is spells is a concerted effort to keep that data under control. As Wendt notes, another of the year’s big trends is cloud storage, which offers companies more flexibility for storing some percentage of their data. I would also add that virtualization has taken a huge leap forward, not only in terms of the technology itself, but also in terms of adoption over the past year. Yet another way to attack the problem.

So if 2009 was all about dedupe for backups, I’m going to guess that 2010 will be very much about data reduction at all points on the data life cycle. What do you predict?

Image: Gizmodo

Lots of special treats awaited the participants of Gestalt IT Tech Field Day. While visiting the offices on November 13 Goutham Rao, CTO of Ocarina Networks stood at the whiteboard and offered a deep dive into the technology behind the company. It was a big hit with the participants. For those who would like a peek under the covers to discover what content-aware dedupe and compression entail, this video is quite a find. Thanks to Simon Seagrave at TechHead for allowing us to repost this video, which he took during the event. We hope you enjoy it.

Ocarina Networks - De-duplication & Compression Deep Dive from Simon Seagrave on Vimeo.

For the entire library of Tech Field Day videos, go to this Vimeo page.

Seems New Year’s is coming early this year. Across the pond in the UK, Ocarina and our partner BlueArc have decided to hand out bottles of bubbly to all and sundry. In short, anyone suffering from a storage hangover–that is, any company that is saddled with more than 20 TB of unstructured data–is eligible for a free bottle of champagne. Talk about the hair of the dog.

The bottle giveaway started last week, and runs through the end of November. It’s open to any companies across EMEA. Here are the basics: send us a sample data set, and we’ll dedupe and compress it by 30-80%, losslessly of course. If we fail, we’ll give you the bottle anyway. (The one thing we don’t accept are cryptologists trying to game the system. This should be a sample data set representative of your company’s main storage repository.) So it’s kind of a no lose situation, unless of course you don’t like champers. If that’s the case, just send the bottle on to this blogger, and she’ll be sure to give it to the needy. (That is, her friends and a few others she’s inviting over for brunch.)

We’re not talking about the cheap stuff here, either. I checked with the team in London, and have been told that it will be Taittinger or better quality.

So, why the sparkly stuff and not, say, a Starbucks gift certificate or some silly piece of shwagola? Because, say our UK marketing team, too much data shouldn’t be a source of misery. It can even (gasp!) be something to celebrate, if you’re equipped with the right tools. In this case, this is a combo of the BlueArc Mercury or Titan and the Ocarina ECOsystem for advanced dedupe and compression. Roll these two together, and you’ve got a delicious mimosa of a storage drink. Slainte!

Here’s how to win: Send an email with “FestiveGiveAway” in the reference line to: ukinfo (AT) ocarinanetworks.com or ukinfo (AT) bluearc.com. EMEA-based companies only, please.

Tech Field Day may be over, but it lives on in digital form–scattered like so many tiny shreds of confetti across the interwebs. One of the delegates at the event, Rod Haywood, put together this video on his Musings of Rodos blog about Day 2 of the event, featuring interviews with Ocarina’s own Goutham Rao, plus Peter Pistek of Nirvanix, W. Curtis Preston of Truth in IT, and Jim Sherhart of Data Robotics.  Rod was kind enough to allow me to repost it, and so here it is for your viewing pleasure:

Gestalt IT Field Day 2 from Rodney Haywood on Vimeo.