Online Storage Optimization

This past year, we at Online Storage Op gathered all manner of images to illustrate our posts. So as a way of looking back at 2009, here are some of the ones we liked the best–and the stories that went with them:

Holodeck fun:

In February, Robin Harris at StorageMojo wrote about a potential breakthrough in storage technology that could change the landscape forever: quantum holographic storage. Online Storage Op was on the scene. It also gave us a chance to upload a pic of a Geordi La Forge doll. Admit it… this is one cool toy.

Squeezing into your Genes:

This blog’s parent Ocarina had quite a year–inking partnerships with a number of major storage vendors and becoming a noted player in the hot dedupe space. It was also the year that genomics labs woke up to the need for better data reduction to deal with the coming onslaught of genetic data. In short, compression can be a matter of life and death. We reported on it here, and our readers got to relive their 10th grade biology class by looking at images like the one above.

Racing for Dedupe

As many pundits are now opining, dedupe really was one of the biggest stories of 2009, not least because of the high profile battle for Data Domain between storage titans EMC and NetApp. In the end, EMC nabbed the dedupe specialist for an eye-popping $2.1 billion.

Booth Babe Mania:

We know our readers are sophisticated types who come here only to absorb information and opinion, and to better themselves for the benefit of all humankind. But for some odd reason we saw a major traffic spike the day we ran our post on the great Booth Babe Controversy. When we asked, everyone quickly told us, “I read the articles.” Mmmhmm!

VMworld a hit

And speaking of images that make storage folks drool, one of the most mesmerizing sights of the year was at VMworld, held in August in San Francisco. Participants descended the escalator to be greeted by gleaming rack of servers and storage–which we later learned was the result of a plan drawn on a napkin by the VMware GETO team. In any case, this year’s VMworld was a major event–and as we rightly noted, it foretold more economic activity in storage and virtualization.

Industry puts aside differences to try to save a life

This is one of the saddest stories of 2009, and one that demonstrates an activist and caring streak in the storage community. When word got out in May 2009 that EMC employee Nick Glasgow was in need of a bone marrow transplant, folks within the storage industry put aside competitive differences and pulled together to find him a match. Sadly, Nick passed away in October. The degree to which he inspired others will not be forgotten.

And, finally…

We never did have an egg and spoon race, but…
In November, Ocarina participated in the first ever Gestalt IT Tech Field Day, which brought independent bloggers from around the world to Silicon Valley for two days of tech deep dives. Our “bring out your data” challenge started tongues wagging well before the event began. Participants brought us their toughest data sets, and aside from those who used archaic encryption software to stump our algorithms, the results were impressive–an average of about 30% reduction on these tougher-than-tough data sets. Plus, the whole event was just a ton of fun. And it didn’t even require that we slog around the mud clapping coconut shells together.

Tis the week for the “out of office” email messages. But the storage blogo-tweet-osphere waits for no man. Here are a few posts that caught my eye this week.

Bas Raayman sees CPU power hitting the wall: The RAM per CPU wall

Rick Vanover says 2010 could be the year for 10GigE - Will 2010 see 10 Gigabit Ethernet go mainstream?

It being the end of a year–and a decade–predictions abounded. We’re pleased to note that when it came to summarizing the top storage stories of 2009, deduplication for primary storage, the specialty of this blog’s parent Ocarina, made the big lists:

Infostor: The top 5 storage technologies of 2009 (and 2010?)

“Storage optimization (or data reduction) technologies such as data deduplication and compression can significantly reduce capacity requirements and costs … Consider data reduction for primary storage.”

SearchStorage - Beth Pariseau: Top 10 enterprise data storage news stories of 2009

“10. Data deduplication branches out. As deduplication settled into a comfortable role in backup, data-reduction technology started working its way into other parts of the data storage infrastructure, including primary as well as nearline and archived data … Ocarina and Isilon Clustered NAS help visual effects studio archive images, cut costs.”

For sheer inventiveness, blogger Stephen Foskett wins the prize with his 2009 predictions post, in which he turns the clock back and takes advantage of 20-20 hindsight: My 2009 IT Industry Predictions.

Meanwhile, social media and tech watcher Louis Gray takes himself to task and looks at all of his 2009 predictions to see how well he fared: My 2009 Tech Predictions: Mixed, But Nailed Real-Time.

OK that’s all for now. Here’s wishing all of you a happy, healthy, green and techy new decade.

Tis the week of Christmas and all through the house, not a creature is stirring except to click their mouse. Black Friday and Cyber Monday are far behind us, but there are still a few doing their last-last-minute online shopping. This entire week, the tubes were getting clogged due to this. Akamai’s Retail Index shows that shopping traffic peaked on December 1st at around 8 million simultaneous shoppers. Retail sites of course are perfect candidates for content delivery networks, who have the scale to absorb the holiday bandwidth spikes.

Here’s a diagram showing the traffic patterns:

When the tubes get clogged, those who make their living selling things on the net can be in real trouble. Several studies have shown a strong correlation between conversion (moving someone from merely browsing to buying) with site responsiveness. Forrester Research recently found that consumers’ expectations for page load time have dropped in half over the last few years, with half of consumers being willing to wait no longer than 2 seconds for a page. Poor performance can have permanent market-share effects: 79% of consumers who had a bad experience on a site are likely to no longer shop there.

Google’s upcoming algorithm changes mean that any retailer with performance problems on its site could be affected, as this will now be a key part of the the mix that determines their all-important Google page ranking. See this interview with Matt Cutts of Google for more information on this.

There are several ways to improve site performance: CDNs move caches closer to users, and good code design can circumvent some limitations of HTTP. If we peer into that traffic, we find that only about a quarter of the traffic is text information, and most of the bandwidth is consumed by images. Therein lies another opportunity to reduce page load times, by optimizing images.

We at Ocarina have been getting to know some of these retailers of late. As you may know, our main focus is storage cost reduction and improving storage utilization. However, we began to run across customers with relatively little storage - at least not enough to constitute a cost or scalability pain who nevertheless were excited about what we had to offer. These were businesses that recognized that our technology had the immense benefit of improved end-user experience.

The typical Internet retailer may have tens of thousands of images (or millions in some cases) in 10-30 TB of storage. These customers use Ocarina’s Native Format Optimization (NFO) workflow, where files are shrunken in their native format, and retain all the savings completely through distribution to the end-user. If we shrink their images by 45% on average, that can literally mean page load times 1/3 faster, which is a huge benefit, not to mention the substantial savings in bandwidth. A happy holiday tale indeed.

‘Tis the season for holly hocks, eggnog, and pundits predicting industry trends for the coming year. Apparently, in the storage blogo-tweet-osphere, getting in the holiday spirit means poking fun at one another.

One of the most notable entries is a series of “Letters to Father Christmas” penned by none other than one of our favorite bloggers, Storagebod (Martin Glassborow). They are broken into seven — count ‘em! — installments (Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, The Last One). Storagebod really kept Santa busy this year.

The letters contain such gems as this jab at storage giant EMC, which he has dubbed “Elven Magic Company”: “there was that V-MAX announcement; boy did that go down well with all of the other Father Christmases. Still, it was great to see you get down with the Intel Gnomes; still a pity that the FAST magic wasn’t ready to go quite yet…”

And this for another company he calls Hippie Pixies (try to guess…) “I think you need a tete-a-tete with your Hitachi partners; they’ve been sleeping on the job! They either come up with a refreshed array or you’re going to have to go acquisitive to get yourself a top of the line Tier 1 array.”

And lest you think that the Elven Magic folk are balking, he got props from the big elf himself, Chuck Hollis. He writes: “He tells all of us vendors what we need to hear, and does it in a gentle and funny manner.  I think Martin has emerged as one of the most insightful and reasonable voices in the storage blogging world, and we all follow him closely here at EMC.”

Martin G. also sends us and our fellows a little gifty: “A special shout out to the smaller vendor bloggers; what you do is very important, more so than the bloggers of the big guys…Most of you do it very well and with a keen sense of humour! Keep it up and keep me smiling.” We’re working on it, Martin, we really are.

But wait, Mr. Bod isn’t the only one getting nipped by Jack frost. Over at his StorageIO blog, Greg Schulz has penned a post for the season: Behind the Scenes, SANta Claus Global Cloud Story. In it, we learn that “The heart or brains of the SANta operation is his global system operations center (SOC) or network operation center (NOC) that rivals those seen at NASA among others with multiple data feeds. The SOC is a 24×365 operations function that covers all aspects from transportation, logistics, distribution, assembly or packaging, financials back office, CRM, IT and communications among other functions…” Wow. For those of us who thought it was just elves turning wooden cranks to pop out Tonkas, this gives us a whole new view of the ops at the North Pole.

And speaking of cranking out wondrous things, Beth Pariseau over at SearchStorage has put together two seasonal posts that provide for highly entertaining reading. What’s great about this is that those of us in the tweetsphere saw her doing at least some of the research in real-time by asking for feedback through Twitter. Here they are:

First, on SearchStorage - What to buy a geek for the holidays

The best quote in this has to be the following from Taneja Group analyst Jeff Boles, “I’ve rejected the Kindle concept. Have you seen the book prices? It’s an entirely disadvantaged arrangement for what is ultimately a compromised experience in the first place. Wait until NEC makes a levitating robot dog that follows you around and holds the Kindle up for you while you do other things…” Quips Pariseau, “We’ll be sure to include this on our “What to Buy a Geek in 2050″ list.” Well, fine but I’m still putting that robot dog on my wish list…

Then this, on Storage Soup: The storage industry’s holiday wish list

In it, is the “wish list” of Wikibon’s Dave Vellante:

• secure cloud
• A dumpster to haul all my backup tapes that I’ve converted to disk-based backup.
• A primary storage device that optimizes capacity without sacrificing performance. (Dave, we may just have a surprise for you under the tree over here at Ocarina….)
• A virtualization performance guru … make it 5 gurus …

Finally, here’s a video showing John Troyer, Community Manager at VMware (and my costar in the Gestalt IT Tech Field Day overview video). In it, we see him open a veritable bonanza of presents from the VMware community. The normally unflappable Troyer cannot help but be blown away by the outpouring of love and truly enviable gifts he received. Those who know John are aware that this is all entirely appropriate to the level of positive energy he brings to that circle of techies. All the way over here at Online Storage Optimization we can feel the glow.

And with that, we wish you all a Merry Merry Christmas, and to all a good night!

George Crump of Storage Switzerland has a post up today on Network Computing that really should be made required reading for every single mid-level executive in the storage industry and beyond. He offers the following simple, straightforward advice:

“…every vendor’s opening slide should have three bullets: The exact problem the solve, why they are different, and how much the product costs. Optionally, the slide could end with a ‘Should we continue? Yes or No?’”

Leaving aside the wisdom of putting a price up on the very first slide, the main point–what problem do you solve?–is the crucial piece here. When you step back and look at it, this is one of those more than obvious pieces of advice. Why else are you giving the presentation? Yet, I dare you to take a look at your slides–the ones you present to analysts, prospects, and others. What does the first one say? I thought so.

If you’re not fixing something that’s broken, why are you in business? Or, if you don’t like the negativity, you can use a phrase that my friend Tony Asaro at Contemplating IT prefers, you’re addressing “pent up demand.” That’s fine too–in fact it might be a more accurate way of expressing the situation.

The main thing is that you’re addressing some THING, not doing a creative but ultimately useless science research project. And most likely you are. But many companies are so enamored with their cool tech they forget all about the purpose of their existence. Their slides (and web sites, and press releases and other promotional platforms) are all focused inward on themselves, rather than on their raison d’etre–their customers.

Here’s an example of a problem that needs solving. Data growth is raging out of control. As it happens, it is unstructured data that is posing the biggest threat to data centers worldwide–email, documents, photos, videos, sound files–in short, the detritus of our wired age. The problem isn’t getting better–in fact, it’s worsening with each passing day. Despite the fact that the cost of disk is going down, the environmental and monetary price to be paid for cooling, power and rack space could cripple a number of otherwise healthy concerns.

Here’s how Graham Hobson, CTO of Photobox, Europe’s largest photo sharing site, explained his problem to me when I interviewed him last spring: “Our data was growing by leaps and bounds–on busy days, millions of photos are uploaded to our servers. If we were to fill a typical data center rack with storage systems, we needed 32 amps of power. But a lot of these data centers in Europe are really only geared up to provide 8 or 16 amps per rack. They were designed for telecom. So there are very few suppliers we can go to that have rack space and that power quota. And of the people that remain they’re not really inclined to give discounts, because their costs are rising.” In short, the company was literally running out of space to store their customers’ data.

That constitutes a serious and pressing issue in my book. What problem does your company solve? Is it spelled out on the first slide of all of your presentations? If not, why not?

Note: This is a special contribution by Eric Scollard, VP Sales, Ocarina.

Japan is the #2 IT market in the world behind the US, so for a growing company like Ocarina it is only a matter of when. not if, we should do business there. Last week, Mike Stoffel, Director of Engineering and I went to Japan at the invitation of JAFCO, the investor who led our recent Series B.  In Japan, JAFCO is very well-known, having funded over 800 companies that had IPOs.

When word got out that they had invested in Ocarina, we got a number of inbound inquiries from potential distributors and end users wanting to learn more about us. In a country like Japan where efficiency and making the most out of limited space is a way of life, the Ocarina message was easy to get people excited about. Having said that, the Japanese market is also conservative and very quality-sensitive. For us, it is important to enter the market at the right time.  This trip was to meet with some prospective partners, bellwether customers and get some feedback on Ocarina and assess conditions.  Front of mind was the condition of the Japanese economy which has been hit hard by the global recession (which they generally call “Lehman Shock” to make it clear who they think is to blame).

Generally, technology adoption in Japan lags North America’s. This past year, 2009, has been the coming out year for Ocarina domestically with adoption of our primary storage optimization solutions across a wide range of industries including: Internet media, VisualFX, Genomics, eDiscovery and Oil & Gas. Given what we saw, 2010 is poised to be the year that primary storage optimization takes off internationally. Despite being generally conservative, the difficult economy and tighter budgets have forced the Japanese IT buyer to think more creatively than perhaps they might normally.  As we often say,” Ocarina is a good solution for bad times.”

Our priorities were to install a couple of customer evaluations and to negotiate a distribution agreement so customers would have a way to buy Ocarina products and to get local, native language support. We were pleased that we achieved both objectives. Without violating any of the NDA’s we signed, I can say that we now have two globally famous brand-name customers installed and evaluating our product and signed a local distributor who will sell and service our products in Japan.  Look for some announcements on these in early 2010.

Mike and I also met with a couple of our storage OEM partners to build some relationships and did a few category-building interviews with the local technology press. The response was very positive across the board.  The initial reaction was generally skepticism that it was possible to compress and de-dupe primary data safely.  Once people understood how our Ocarina ECOSystem assured data integrity while reducing data footprints by 30%-80%, they wanted to check it out with their data.  We will probably be back there again in a few months and hope to make some exciting announcements at that time.

Two identical photos placed side by side. Both are are blown up to large size on the wall. The difference? One has been compressed by 70% using a specialized compression technique called NFO, while the other is the original. Can you tell the difference?

This was the challenge we offered participants at the recent Gestalt IT Tech Field Day. Calling it the “NFO Challenge,” we at Ocarina asked the attendees to pick the optimized photo from a series of image pairs. The participants in the event were some of the sharpest folks around–a group of independent tech blogggers from around the world–but we stumped them. In fact, the winner of the contest only guessed 50% correctly. It was that difficult.

NFO stands for “Native Format Optimization.” It’s a workflow option that Ocarina provides for image and video customers to shrink their data while retaining the native file format after optimization. Today, George Crump has a thoughtful piece, sponsored by Ocarina, that gets into detail about the technology on his Storage Switzerland blog.

Here are the basics: a JPEG comes in, a smaller JPEG goes out. Right now we’re supporting JPG and GIF, and we have beta support for PNG and h.264 video. Although our GUI offers a “volume knob” (which goes to 11 by the way), the standard settings shrink media files by 30%-50% with no perceivable quality loss.

To be clear, this is lossy compression in the technical sense. There’s no substitute for looking at the real thing. We’ve had plenty imaging experts at our customers tell us that the changes made by our algorithms really don’t impact quality. But if you need bit-for-bit lossless compression, we provide that through our standard compression offering.

NFO optimization introduces image changes that render an image that is visually identical. The most common method for validating this is to print some really big before and after images and go through them in excruciating detail to look for artifacts. You can calculate PSNR and other quantitative measures, but there’s no substitute for looking at the real thing.

The customers that have really taken off with this technology include folks not so much concerned with storage, but companies trying to save money on bandwidth. Internet photo sites are an obvious sweet-spot, but some less obvious users are large Internet retailers, whose bandwidth bills are dominated by product imagery. These guys are really excited to start building out with Ocarina. And that’s just the economic benefit of NFO. The ultimate benefit is improved end-user experience and loyalty when an image-laden web site loads faster. Now we’re adding some serious value to Internet brands.

Here’s the secret sauce: Ocarina’s software looks for ways re-encode the media in ways that align image encoding with the sensitivities of the human visual system. We’ll generally analyze the image in the pixel domain, making adjustments in de-noising, Luma/Chroma quantization, Huffman optimizations, better motion compensation, and more.  To read about these optimizations, and hear more about what our team of Ph.D.s has accomplished with these algorithms, read our new white paper at OcarinaNetworks.com (http://www.ocarinanetworks.com/images/resources/nfo.pdf)

It’s been a tough year — a worldwide recession, a sluggish housing market, rising unemployment … and on top of all that, the tarnished image of one of sports’ most squeaky clean players. Well, actually, there have been some bright spots. As DCIG blogger and storage analyst Jerome Wendt notes while looking back at the past year, “Deduplication is the Big Success Story of 2009.”

Wendt writes: “Deduplication is arguably one of the most notable trends of 2009 as it has been widely adopted by users after bursting onto the scene just a few years ago and has grown to be included in both software and hardware products.”

Wendt focuses on dedupe for backups, where there has been much publicized activity over the past year. The big storage story of 2009 was of course the battle between storage titans EMC and NetApp over backup dedupe specialist Data Domain. He cites an industry survey from SearchDataBackup that indicates that 41% of enterprises either are or are seriously considering dedupe to control data growth and costs. He also notes that the despite the predicted demise of Quantum, that dedupe company remains strong.

Dedupe for backups is one part of the cost reduction puzzle. Another part is to reduce data at the source, in primary storage. This is of course the specialty of this blog’s parent Ocarina, which implements a unique combination of content-aware dedupe and compression to achieve startling results. It focuses on the very types of unstructured data that are driving storage growth today–emails, images, documents, and so on. The company has been partnering with almost every leading storage provider, including HP, EMC, HDS, BlueArc, and Isilon. Another  leader in this space is NetApp, which has a strong dedupe for primary offering that has also garnered a great deal of attention.

Here’s the thing, the economy might be slowing down, but data growth continues apace. This is one reason that the storage industry has been thriving this year. But rather than standing still, what is spells is a concerted effort to keep that data under control. As Wendt notes, another of the year’s big trends is cloud storage, which offers companies more flexibility for storing some percentage of their data. I would also add that virtualization has taken a huge leap forward, not only in terms of the technology itself, but also in terms of adoption over the past year. Yet another way to attack the problem.

So if 2009 was all about dedupe for backups, I’m going to guess that 2010 will be very much about data reduction at all points on the data life cycle. What do you predict?

Image: Gizmodo

Dell’s Inside Enterprise IT blog has identified “10 Trends to watch carefully.” The post is from the Gartner Data Center conference that wraps up today in Las Vegas. One of the biggest and most important trends? The coming “data deluge” that will pile onto company IT departments like a load of bricks.

Over the next five years, enterprise data growth will increase by a whopping 650%. And here’s the kicker: 80% of this data will be unstructured. That means emails, documents, photos, and all the other files not in databases. The answer, according to the experts: attack with virtualization and deduplication.

Might we also suggest that a combination of content-aware deduplication and compression would yield even better results? The modern enterprise is dealing with all manner of data. These are the types of files that often stymie traditional block-level dedupe. What can it do with images, video, audio–not to mention compound documents such as PDFs and Zip files? Often, very little.

As we showed at a recent event, Tech Field Day, Ocarina reduced the toughest data sets by an average of about 30%. (Final results will be out soon.) In fact, the only ones that stymied the system were those that were deliberately encrypted using unusual or outdated methods–not a typical use case to say the least!

What this means is that over the next few years, the flow of data will not only increase, but it will become far more complex to handle. If you think about the speed of innovation, there’s a strong chance that there will be files we aren’t even aware exist yet. What do you think? How is your enterprise handling unstructured data now? What will it do differently in the next five years? Comments encouraged!

Two items jumped out at me this week. First, Francine Hardaway’s post on being an early adopter, “Do You Write in the Cloud?” and second, this New York Times story about Cormac McCarthy’s typewriter.

Hardaway, a serial entrepreneur and founder of Stealthmode Partners talks about how she and other bloggers compose their text. Mashable’s Pete Cashmore, along with Buzzmachine’s Jeff Jarvis and Gina Trapani all write in the cloud. Hardaway either writes directly in Gmail or sends one to Posterous from her Gmail account. If there is a bad wifi connection, she uses Evernote and synchs it to the cloud before moving it to Wordpress. (Am I the only one who composes directly into Wordpress?)

The main point here is that Hardaway illustrates how she was ahead of the curve each time, and was able to show successfully that you could still think and write using the latest “newfangled” tech–whether a manual typewriter or a word processor. Hardaway recollects:

“…50 years ago, I started composing on a manual typewriter when most other people composed in longhand and transcribed.  Many famous authors wrote in longhand their entire lives. People thought I was crazy for going straight to the typewriter. But I’ve gone seamlessly from the typewriter to the word processor (we could dictate into a  Digital Equipment Corporation word processor in the Maricopa Community Colleges in the 70s, when I was a professor.)  They thought I was crazy for dictating instead of typing when I did that.”

All of this was sloshing around in my brain when I came upon the aforementioned story of Cormac McCarthy’s typewriter. McCarthy is the author of a number of bestselling books, including the recent “The Road“–now a movie, and “All the Pretty Horses,” among many other award winners. How did he compose his text? On a portable Olivetti typewriter he bought at a pawn shop in 1963. He moved on from long hand, but after that he just stopped.

We’re talking about two ends of the spectrum. The early early adopter. And the … well, I don’t think we can call Cormac McCarthy a late adopter now can we? Of course, most people are somewhere in the middle. Enterprises are managing an influx of data that is based on what would be considered the median behavior–a need to store all manner of files, from Office documents to family snapshots to the latest Lady Gaga download. Oftentimes, the response is to try to ignore the problem. More and more files are shoved onto expensive primary storage. Suddenly, the problem reaches massive proportions and emergency measures are taken, rather than a well thought out tiering/deduplication plan or a move to a reliable cloud storage platform. But I digress. The main question here is, what kind of adopter are you? And what are the strengths and weaknesses of each approach?