Online Storage Optimization

I just got back from the Exploring Next Generation Sequencing conference held at the Rhode Island Conference Center in Providence, RH. I was pleased to encounter a number of folks who were extremely enthusiastic about the Ocarina solution for Life Sciences. In a way, I shouldn’t have been all that surprised. We’re the only vendor in the industry to have delivered industry-specific data-reduction solution. In other words we understand their file types: SRF, TIFF, FASTA, etc. and therefore have the power to reduce those files far more effectively than other data reduction options.

It’s a very exciting time for the entire Life Sciences industry, which is in a period of rapid advancement that could soon translate to cures for some of the most deadly diseases of our times, such as heart disease, cancer, and Parkinson’s. This is mainly due to the fact that there have been quantum leaps in gene sequencing computing that enable genetic data to be analyzed to levels that weren’t possible even a few years ago.

As I mentioned in the Luncheon Keynote that I delivered, the genomics space is in many ways parallel to film-making circa 1998. (See our recent interview with animator Mike Huber.) There are a few early adopters with big capex investments, but the digital workflow is really just emerging, and has yet is only narrowly deployed in the commercial marketplace (pharma, healthcare, agriculture, etc).

Here are some of the facts about this industry that made me sit up and take notice:

- 80% of all sequencing is performed as part of research and 20% as a commercial effort.
- We still see strong investments in vertical integration: For example, Complete Genomics is designing a scanner for their internal use only.
- Thus far, only 12 people have had their DNA completely sequenced.

- No one knows what the variation is across the human genome, or what the “normal” human sequence is.
- There are already efforts to resequence known genomes to improve accuracy to acceptable levels.
- Entire fields of genomic research are just emerging: Genotyping, large-scale correlation (GWAS), Transcriptomics and RNA, agriculture, genealogy, pharmacogenomics, elective sequencing, discovering the microbiome, etc.
- Within 2 years, the cost to sequence an entire individual’s genome will be less than the cost of many specific chromosomal deficiency tests today.
- Within 3 years, some hospitals will generate a sequence for every incoming patient, partly mitigate the risk of adverse drug reactions.
- Only an extremely small fraction of physicians have the knowledge to meaningfully leverage genomics in caring for patients.
- Life Sciences storage per employee has grown 500% year-over-year [Corporate Tech survey].
- Sequencing throughput in 2005 shifted from linear (2-3x growth per year) to logarithmic growth (10x per year), leading to dramatically reduced cost per sequence and anticipated explosion of usage models. [Duke University preso]
- Existing sequencer models use still-image based data acquisition (typically 8-14bit TIFF images), which are commonly thrown away to relieve storage load.
- Next generation sequencers from companies like Pacific Biosciences will use video-based data acquisition.

The conclusions you should draw from these factoids are the following:

1) This is a highly immature industry (in the MBA sense, not the people sense!) with dramatic market growth anticipated as adoption begins to take off in non-research markets, and;

2) Data growth at any given customer will be nonlinear and dramatic, with the cost of storage infrastructure taking an increasing share of the IT budget.

Clearly Ocarina has an important role to play in the sequencing workflow, not only to reduce storage consumption for inactive data, but also simplifying and improving data packaging, movement, replication, and backup. In short, there’s going to be a lot of opportunities to add value. Fortunately we’re getting a lot of good feedback from the industry leaders like Illumina and Roche/454 to keep us working on the right things. Stay tuned and come see us at Bio-IT world Oct 6-8!

Friday’s CNBC segment is worth a watch if you’re wondering where the greatest growth will be over the next decade in the technology sector.

“Everything that we are using now–videos on YouTube. Everything we’re doing on our phone. On and on. It’s all about storage and putting stuff on the net,” opines Roger Nusbaum of Your Source Financial, who adds, “Cloud computing comes into play here.”

This week, the New York Times Bits blog reported that a winner has been announced in the $1 million Netflix challenge. The prize went to the team that could come up with a better “predictor” of customer behavior than the company’s current in-house software, Cinematch.

It’s more than a gimmick. Having a reliable and effective tool that will tell whether a person is likely to prefer “Sleepless in Seattle” over “Napoleon Dynamite” on a given Friday night represents the holy grail for the mail-order movie rental company, which must continually come up with ways to satisfy its customers’ need for entertainment.

Many have been anxiously awaiting the results. What’s most notable about this prize is that it has turned out to be a great example of the power of teamwork in innovation–to a level far beyond what had been previously imagined.

In fact, as one team approached the finish line, it became a race to aggregate other teams in an attempt to develop the winning algorithm.

The Times quotes Chris Volinsky, a scientist at AT&T Research and a leader of the winning “Bellkor” team, saying that the blending of different statistical and machine-learning techniques “only works well if you combine models that approach the problem differently … That’s why collaboration has been so effective, because different people approach problems differently.”

Reading this, I couldn’t help but notice the parallels between this and what we are accomplishing at Ocarina. In fact, the intensely collaborative approach is what makes Ocarina such an exciting place to work. We bring together great minds in order to yield algorithms for data compression. In addition, compression is as thorny a problem to solve as consumer behavior, and is similar in ways that might surprise you.

You can work on having one good generic compressor – and there are many good free generic compressors out there – but the best results will be achieved by having a set of more specific algorithms and an infrastructure for tying them together to get the best results. What’s more, compression, like the Netflix challenge, is all about prediction. If you can predict the next value in a data set, you can compress it. So the approach that produced the best results for predicting consumer rental behavior are a good signpost for how to go about getting the best prediction on how to shrink a data set. This doesn’t mean the same algorithms, but the same concept of having multiple algorithms, a content-mixing and adaptation approach, and tying things together to get a whole that is greater than the sum of the parts.

Let’s take a look at a very well-known and high-quality generic compressor, LZ (and its many variants, which include: LZO, libz, glib, etc). The term “LZ” comes from two pioneers in compression algorithms for computers, Abraham Lempel and Jacob Ziv. LZ is based on their work, and has evolved quite a lot over the past 30 years since they were first developed. This family of compressors will try its best on whatever data pattern you throw at it. If the data pattern is uncompressed, LZ variants will do quite well. Plain text, numeric data, and other application data that has open patterns will be well-compressed by LZ.

As powerful and useful as it is, LZ has its limitations. Notably, it is stymied by file types that have already been compressed by the application (or by some other tool), as well as files that are compound documents with multiple different data types inside, and media files. Alas, more and more of today’s files are either office documents (MS Office, PDF) or media files (photos, video, special effects) that are compressed in some simple way by the application that created them.

Compression, as I said, is all about prediction. A compressor builds a dictionary or context, looks for patterns, and if the pattern repeats or can be predicted, then it can be replaced with a shorter symbol.  The longer the pattern you can find, the better then win for replacing it with a symbol. The more patterns you find, the better results you get.

The trouble is that when one compressor has gone through a file, it has taken out at least the low-hanging fruit. If it’s a good compressor, it may have done a very good job of finding patterns and replacing them with efficient codes. Video – which has been an area of intense compression research for years – is a good example of this. An h.264-encoded video file is going to have had the easily compressible stuff already well taken out. Photos, graphics, animation, and other media-related files are similar – applications from PhotoShop to Maya (and inferno, Flint, and Flame), and most sound file types (WAV, mp3, FLAC) will already have been compressed. This often will defeat even a good generic compressor like LZ. The patterns it looks for will have been obscured by the compressor that processed the data first.

You can see this easily by running LZ against a directory with a wide variety of file types.  Put some text, some spreadshseet, some photos, and some other media files in a directory and see the percentage compression you get. There will typically be a wide range. This does not mean LZ is not a good compressor; what it means is that it seeing a mix of files it can compress and ones it can’t.

So how do you get better results? Not by building a better generic compressor, that’s for sure! Rather, by getting specific. This means building a family of compressors, each of which is aware of specific file types and their data patterns. This is the approach we have taken, and the results are so remarkable that our customers — whether in the Hollywood post-production industry, life sciences, Internet media, or some other industry–  are often stunned by the difference these algorithms make when it comes to reducing their data.

All of which is to say that the Netflix prize researchers’ discovery is much the same as ours: the winning strategy is not going to be one better compressor, but a cooperating framework of compressors that work together. Teamwork, not just among people but among interlocking technologies, is the difference between good and great — between generic and extraordinary.

In what seems to be a theme this week in Online Storage Optimization–we’re all about what’s next, and what’s “out there” in the stratosphere. We’re also not beneath making as many pop cultural sci fi references as we can. That’s just the way we roll.

Analyst George Crump had a nice piece in Networking Computing (the new umbrella publication that has eaten Byte and Switch for breakfast). His article, “Deduplication Moves Beyond Deduplication” gives a good overview of the state of dedupe and where it might be headed. For those of you who have been reading this blog regularly, we’re predicting a major leap in innovation around dedupe.

We see an end-to-end or “global dedupe” as the obvious next step for this groundbreaking technology. As Carter George, our lead writer noted a recent post: we see a future in which dedupe will have post process architecture, and will find duplicates across multiple nodes and multiple storage pools. It will also integrate a combination of technologies–most likely the best will be those that have both dedupe and content aware compression. This has been the direction that this blog’s parent Ocarina has taken, and it has proven a very wise strategy.

As Crump notes, Ocarina differs from other solutions in that it “adds deduplication to the process as well as content specific optimizers that provide a greater understanding of the file formats being processed.” The next wave of dedupe, he predicts, will still involve two or three different solutions within an environment. No doubt there will be a path that takes us from here to the ideal of simplification we envision. Meanwhile, they will continue to add features that ensure that the investment in dedupe is amply leveraged.

I’d say that one great example of that is the ability to easily facilitate migration among multiple tiers of storage–in other words, migrating data and tracking its location as part of the optimization process. A solution that can do that in addition to reducing file sizes offers notable savings and simplification on both ends. Oh, did I mention Ocarina already does that?

Life can be tough for a Jedi Knight. And in the UK, it can lead to run-ins at the cereal aisle. In one of the more retweeted and blogged about stories of the week, the UK Guardian reports that Daniel Jones, the 23-year-old founder of the Church of Jediism, was told to remove his hood while shopping at the grocery chain Tesco.

Jones, who also goes by his Jedi name ““Morda Hehol,” has lodged a complaint with the store, alleging religious discrimination and is calling for a boycott. The hood he wore, he said, is part of his right to exercise religious freedom. The Tesco empire has “struck back” (the Guardian already made this joke but anyway…) claiming that they respect the religion, but respectfully ask that he remove the headgear.

Quoth the Guardian: “…Yoda, Obi-Wan Kenobi and Luke Skywalker - all appeared in public without their hoods…”

This is just the kind of situation that makes me wonder what the galaxy is coming to. I mean really, if you’re a Jedi, wouldn’t you just do a few mind tricks on Tesco security until they’re convinced that this is just the type of fancy dress they’ve always been interested in seeing in their supermarket? I was also deeply disappointed to hear that to prove his Jedi status, he gave them a (get this!) business card.

As someone who knows the power of using the force for all kinds of maneuvers, I can say that the old “here’s who I really am” trick only works if you’re using magic paper a la Doctor Who. Otherwise, it screams “stuck on the third rock without my airspeeder” with the power of a thousand sun crushers.

We here at Online Storage Optimization will be following this story closely, reporting any breaking news that comes over the transom whenever and however we can. Meanwhile, may the force be with you, whether you’re clipping coupons or defeating the Zann Consortium.

Image of cool origami Yoda from: http://helenkosings.wordpress.com/2009/03/28/a-crazy-week-a-surgery-date/

Fall is practically upon us. Those of us of the Jewish faith are slicing up the apples and dragging out the honey bears while listening to well-wishes from the prez. And for everyone, it seems that a sense of renewal is in the air.

Some items this week caught our eye over here at Online Storage Op:

You Lie! Who Coined the Term Data Deduplication? - Dr. Dedupe over at NetApp goes all “history detective” on us, and tries to discover who first used the term that has come to mean so much to so many — “data deduplication.” A fun read for a Friday.

Marc Farley skewers Chuck Hollis - 3Par blog-o-riffic Storagerap is now featuring comics and I for one am shaking in my boots about being lampooned. You could be next, so if you see a guy with a camera strapped to his steering wheel shadowing a cherry Vega, be warned. You heard it here first.

Chuck’s blog has a really interesting post on “Managing Your IT Career in Turbulent Times” — This is not the one Farley skewered but the one before it. A great example of a thoughtful, helpful post from a senior insider at EMC. It gives some great specifics on trends.

Not storage-related, but anyone who hasn’t been living under a rock lately must have noticed the complete Kanye mania that has taken over the Internet, and… IMMA LET YOU FINISH BUT FIRST I MUST TELL YOU THAT KANYE HAD THE BEST INTERRUPTION EVER! … Ahem! … Enjoyed the endlessly circulated YouTube video? Now buy the t-shirt.

We wish you all a healthy, happy weekend and we’ll see you back here next week.

For those who are new to this debate, I will catch you up. First, there was VMWorld–which was kind of like a cross between a high level MIT graduate symposium and Ringling Bros. Barnum and Bailey Circus (with strippers). Whatever the intentions of the organizers, somehow that show was a strange concatenation of labs on the latest virtualization technology and a show floor dominated by jugglers, comics, and models in sexy nurse costumes.

In response, Stephen Foskett wrote a SiliconANGLE post urging: “(Don’t) Make Your Startup Look Stupid with Booth Babes and Chotskies!” He raised the question - what do these cheesy attention-getting tactics actually say about your confidence in your company’s offering?

But then F5’s Kirby Wadsworth came along and reiterated his belief that booth babes are a great way to get people’s attention at trade shows. The two of them went mano-a-mano on the Storage Monkeys podcast, Infosmack. Oh yes, and this blogger was also on the line, smacking both of them down.

But our story does not stop there. Mr. Foskett is showing himself to be willing to take his original argument to the limit. As he revealed yesterday, he is going to do something utterly radical for his next trade show appearance in his booth with Nirvanix. It will be entirely plain and simple. No attention-getting tactics, cheesy or otherwise. It will be the “Dogme 95” of booths. I for one am very curious to hear how well this works–it could change the face of trade show marketing spends forever. To be continued…

There’s a really interesting post up this week by Danny Sullivan of Search Engine Land about how he blew through his gmail space limit. Hardly the usual experience–most of us are using only a small percentage of our overall space. But it happened to him, and made him stop and think. He became suddenly aware of the storage reality of email– all the energy and disk space we are using to store all manner of “crud” that goes through our email boxes.

“Is there a case for email conservation?” he asks.

For those of us who are steeped in this stuff, it’s great to see someone take the experience apart as he does. The discoveries he makes are pretty mind-bending. For example, he calculates that Google is adding storage at about 1 MB a user every three days to keep up with demand. To deal with his own email, Sullivan realized he could take actions such as turning off Twitter and FriendFeed alerts, clearing out spam, and deleting Google analytics alerts.

We know first-hand at Ocarina the reality of what it means for a photo sharing site to offer, for example, free, unlimited storage. In short, it means they’ve got to figure out a way to compress or otherwise reduce those files, as well as to ensure they’re properly tiered and managed, or they will literally run out of places to put them. (Read our Photobox case study with Isilon for that story.)

As it happens, I recently had a conversation with a friend who was angry about the fact that her ISP had deleted all of her emails from three years ago or older. She claimed that they hadn’t warned her about this, although my guess is that they did and she didn’t see or understand what they were telling her until it was too late.

I explained to her that from her ISP’s point of view, these were files that had remained untouched for a very long time. Why should they have to keep storing them? I then went into a probably rather long-winded explanation of what I understood about how those types of files are being stored by various services. Google has its own propreitary servers/storage, and they along with a handful of others do single instancing, which cuts back the amount of space, obviously, but still comes down to the reality of storing the files.

“Wait a minute!” she suddenly asked. “Are you saying that each one of emails is being stored somewhere?”

OK all you storage geeks who are reading this, you can laugh now. But really, this isn’t all that surprising in light of the way it’s portrayed. As Sullivan points out in its post, Google actually advertises that you never need delete an email. And as he discovered, this turned out not to be strictly true.

Still, it’s amazing to me how many of my non-techy friends are surprised to learn that every piece of information they create on the Internet must live somewhere. It’s hard to say what they were imagining up to that point, but most of them seem to view the Web as a sort of magical place in which images, videos, documents, and other useful things simply flash into existence when needed and then disappear into some sort of ethereal never-never land the rest of the time.

They are often shocked and not a little horrified to learn that those files are online at all times–spinning away on disks 24/7, using immense power, space and cooling resources. This is hardly the clean, green internet that most people imagine.

Email is a particularly tricky one to grasp. While it’s true that MS Outlook Exchange is on a database model–arguably a more efficient way of managing email files–all of the highly popular web-based email programs (Google, Yahoo, etc.) treat each individual email as a separate file. Attachments, which often make up the bulk of the data, are deduped at the file level in many cases. That is, if I send a PowerPoint to a distribution of 20 recipients, my Gmail program will recognize that document and single instance it. However, if I take one slide out of that PowerPoint and replace it with another one, my email service will not recognize this change, and will instead see it as an entirely different file. With millions of emails bouncing around day and night, this adds up to an awful lot of data.

We encourage you to ask your friends and relatives to find out what they think — or what they think they know — about the way things work. You may be surprised, and they might learn something!

Bas Raayman, an SAP consultant has a post on StorageMonkeys that questions the emphasis on de-dupe and thin provisioning, when evidence is mounting that storage at many enterprises is largely underutilized. This got me thinking.

Bas is asking a good question: Why bother shrinking your data when you’re only at 50% capacity? Now the premise is debatable, as certain verticals such as life sciences, media and entertainment, social networks, etc. are as prolific as ever and driving a lot of the 2009 revenue of our NAS partners Isilon, HDS, BlueArc, HP, and EMC.

However, for customers where utilization is low, it’s legitimate to ask the question “why mess with de-dupe?” The answer lies in understanding the resources saved beyond just storage capacity.

Here are some obvious opportunities:

1. Higher performance data migration, backup and replication (thus improving reliability and reducing bandwidth costs);
2. Consolidation into fewer filers (thus reducing management cost and OpEx);
3. For Internet businesses, reduce distribution and CDN costs by delivering smaller files.

Of course end-to-end optimization is a pipe-dream for most vendors where de-dupe is a low-level embedded block-based solution. These point solutions lead to the problem of “de-dupe headache” (yes, you heard it here first!) where islands of disparate de-dupe solutions work in isolation, and any data-movement requires re-hydration and re-shrinking into the next solution. This is something that Curtis Preston takes up in a recent article in SearchStorage.

Here’s where I think an out-of-band content-aware tunable optimization solution like the Ocarina ECOSystem has bridged the gap. It delivers end-to-end fully optimized workflows--not just storage, but bandwidth, power, and reliability. This is really the end-game and where we’ve set our sights.

One example–Ocarina has been shipping the capability to optimize files while leaving them in their native format. In other words, no reconstitution mechanism is necessary. This is particularly useful for photo sites where JPG images are optimized using our visually lossless algorithms, but still retained as JPG files. So the photo site can distribute shrunken images without quality impact. We may not deliver the 8x reduction of a bunch of office docs, but 30-50% optimization delivers massive benefits in bandwidth cost in addition to storage savings.

So even for storage consumers with low utilization, it’s becoming more clear that end-to-end data optimization will lead to improved costs across the entire workflow and infrastructure. In short, as the headline says, it’s not just about storage anymore.

HDS has had an interesting run when it comes to its social media efforts. With Hu Yoshida, they’ve got a heavy hitter putting out blog posts with gravitas. This hasn’t stopped them from adding several more bloggers, including consultant Tony Asaro who offers a refreshing, semi-outsider’s point of view. As of last Friday, Miki Sandorfi, HDS’s chief strategist of file and content services has joined their clutch of storage writers. He penned a post yesterday outlining the the new HDS High performance NAS products: HNAS 3080 and 3090, that is definitely worth a read.

And speaking of HDS, in case you missed it, Ocarina and HDS announced an integrated product this week. It’s available immediately–a boon to HDS customers looking to reduce their data and maximize disk capacity.

Here’s the full news release:

Ocarina Networks Integrates with Hitachi Data Systems

Ocarina Networks Integrates Content-Aware Compression and Dedupe Solution With Latest Hitachi Data Systems NAS Family Offerings