The acquisition of Ocarina by Dell has now closed, and we can now say some things about where we are headed.  Mergers and acquisitions can often be a time of confusion, as the new combined company figures out what the new plan will be.

I’d like to use this first blog after the closing of the deal to clear up some of that confusion.    Some of our competitors have tried to take advantage of the period between the announcement of the deal and the closing (a time during which neither company can make forward-looking statements about joint operations) to sow uncertainty and doubt amongst some of Ocarina’s most-valued customers:  our OEM and resale partners.

Now that the deal has closed, I can say that Dell is actually very interested and committed to pursuing OEM partnerships for the Ocarina technology with other storage vendors.

One competitor in particular tried to imply that our OEM’s would have to drop Ocarina and go looking for a new dedupe technology.

This is simply not true.  While each partnership is a unique case, in general, the OEMs that want to continue doing business with us will be able to. What’s more, this is now a stronger solution, both technically and on the business front.  Dell plans to invest in growing the Ocarina engineering team – and provide a richer roadmap and have a faster cadence.    Clearly, onsite support will now be available worldwide for those partnerships that involve reselling an Ocarina-based dedupe appliance. Software support will eventually be available in many more languages, also around the clock. In addition to those obvious technical benefits, some of the risk of doing business with a startup is now also resolved.

Finally, I think that Dell sees value in an ecosystem of compatible and complimentary dedupe products. It’s great, for example, that Ocarina dedupe can work together with Commvault and Symantec dedupe. We think it is good for customers to see that compatible ecosystem grow. What is good for customers is also good for our OEM’s – by participating in an open ecosystem that gives customers not only the best-of-breed technology, but also a rich set of choices for deployment, they are strengthening the solution they offer and the story they have to tell.

What was even more impressive about this survey was the respondents themselves. While it’s easy to understand for the largest companies with their heavy data sets that scalability is a major strain, two thirds of these respondents reported having less than 50 terabytes of storage, meaning these challenges have now hit mid-size businesses as well. The <50TB crowd is going to be more risk averse, and reducing their storage from 3 arrays to 1 array isn’t necessarily the “killer” value proposition. The value proposition of compression and dedupe technologies includes things like reduced backup windows, which can only be obtained by a solution that supports end-to-end benefits, proves to customers that it’s trustworthy and doesn’t carry additional administrative burden.

Also intriguing was the comment by many respondents that the growth of storage was in transactional applications (database and e-mail) rather than unstructured data. Many of Ocarina’s clients in multiple market segments are seeing dramatic growth in unstructured data, but in this business tier, more traditional applications are adding to the data glut. Also, there is little doubt that in a short amount of time VM applications will appear as part of that category. This is also in contrast to the “Petabyte vertical markets” which are chock full of unstructured file data, and a good lesson for OEMs to take away; any embedded dedupe implementation needs to deliver results in structured-data applications, and that’s a non-trivial requirement. Common dedupe solutions in the market today would create noticable performance impact on these apps, which is why Microsoft removed single-instancing as a native feature from Exchange. A dedupe solution in these applications needs to include workflow and application awareness, for example compressing and deduping only those portions of the database (or VMDK) that are inactive. Ocarina calls this Heat Index Management, and it’s a feature included in our ECOsystem for OEMs.

As industry observers have seen in the last several years, backup has been the killer driver for deduplication – the first wave, if you will, because the ROI of D2D backup is tremendous with this feature in place. But in the survey there is strong adoption (>50% of deployments) of dedupe for archival and other non-backup apps. Of course this isn’t unexpected to Ocarina, because we’ve helped drive this shift towards compression on primary storage and the entire data lifecycle.

As data growth continues in all markets, we believe strongly in the need to compress and optimize across the datacenter, from primary storage to backup and archival. InformationWeek has done a fantastic job demonstrating real customer demand for these technologies, which we expect will increase. If you haven’t walked through the presentation, make sure you do at http://www.informationweek.com.

When I heard the news yesterday that Dell would be acquiring Ocarina Networks, my first reaction was pride. I couldn’t help but give myself an inner “high five” for having recognized what a great company Ocarina is and would become. My second was to congratulate Dell on their smart choice. I was lucky enough to have worked with the people at Ocarina, and now a whole new group of folks will have this opportunity.

The people at Ocarina Networks are not run-of-the-mill. They are not even above average. They are extraordinary. Anyone who wants to understand what makes a technology company excel would do well to study them. As I wrote in my final, farewell post, Murli Thirumale, CEO of Ocarina is about as far from the image of the typical start-up CEO as you can get. He is soft spoken, thoughtful, and a good listener.

He built his company around a well thought out business philosophy. He first identifies a problem or need, and then designs a product in response to it. In this case, the problem he identified was the proliferation of data worldwide. Rather than go for the obvious, he brought together a team to research the problem and solve it in a new way. This approach is rare. Most of the time, you have start-ups that try to launch based on a technology they themselves developed–in other words, their own pet project. It takes discipline to go about things the way that Murli does.

Murli also showed himself to be a leader in the true sense. Rather than feed his own ego, he chose to surround himself with giants of the storage industry, and he allowed them to get the job done the way they saw fit. Carter George, VP Products, is one of those people. I am particularly indebted to Carter. He was truly a mentor to me. Despite the intense demands of his role at Ocarina, he always had time to answer my questions. He never treated me as anyone other than an equal. I also got to know Goutham Rao, the visionary genius who is the company’s CTO. In addition, I met Dave Withers, who is the man behind the company’s multiple partnerships with top storage companies. For the last few months of my tenure, I worked closely to ramp up the social media program with Mike Davis, director of marketing. Dell is extremely fortunate. They are acquiring technology, but to me the real gift is that they get to work with these remarkable people.

As the headline says, this to me seems like the beginning for Ocarina. It is a bold plan to take the vision of end-to-end dedupe and make it real. This is the next step for the storage industry–and a crucial one if storage costs are to be kept in line, and infrastructure is to keep up with the demands of the real world. We are living in a time when data growth is spiraling upward at rates that no one could’ve imagined even a decade ago. This is the time for a cohesive, meaningful response to this reality. I couldn’t imagine a better outcome for Ocarina, Dell, and the industry as a whole. I am honored to have been a small part of it.

For our people, it is a great outcome.  Good people build great technology.  Great people build products that get used over and over again by a large number of customers.  Getting our world-beating dedupe solutions into the hands of a large number of customers rapidly is what we are all about.

We have been working with Dell for over a year.  The people we have met all the way from engineers up to Michael Dell are intellectually curious, direct, driven to succeed,  talented and fun to work with.  In fact, they are a lot like us at Ocarina! Culture alignment?  You bet baby! Finally, even with its success, Dell’s storage business is relatively new and their willingness to fund it (they bought us didn’t they?), long-term expansion plans and focus on talent acquisition ensures a great opportunity for the talented team we bring across all functions to Dell.

Primary dedupe leadership by the year 2012.  Now coming to you at rocketship speed powered by Ocarina AND Dell.  Watch out storage industry!

Here are a series of requirements that we’ve consistently heard from OEMs in our discussions, and here’s how Ocarina addresses those in the ECOsystem for OEMs.

1) The solution shrinks data well.
Ok, so this is obvious. Dedupe is supposed to shrink data, and that improves system utilization and reduces costs. Doing this well is about algorithms, and Ocarina has been on top in the algorithm game since we started shipping our primary storage optimization products almost 2 years ago. We’ve shown in some deployments that we can deliver 80% savings when traditional block dedupe delivers no more than 30%. We believe in fact that we’re the only storage technology vendor to actually employ algorithm PhD’s whose sole job is to invent better algorithms…which is something that has proven really useful in specialized markets where a few unique content types dominate the terabytes.

We are the only provider today that deploys dedupe and compression concurrently. Our dedupe algorithm is a content-aware, variable-block, sliding-window approach. ‘Content-aware’ is an overused term for sure, but here it reflects for example that we’ll recognize monolithic data structures in a stream like a JPEG blob, and we know that slicing that JPEG into 8KB chunks for dedupe delivers absolutely no benefit, and thus is a complete waste of time, CPU, and memory. We’ll treat that JPEG (and other data types like it) as a contiguous chunk, which makes our dedupe namespace extremely fast and efficient.

2) The solution minimizes time to market.
Most OEM vendors are under considerable time pressure to bring dedupe to market, either as a competitive response, or because their customers are demanding it. But the OEM’s dev and test engineering resources are always limited. With that in mind we made a point of developing a full featured library. That means not only do we slice the data and do hash lookups (as with the Permabit product), but we also do the dedupe, compression, on-disk data management, metrics and reporting, throttling mechanisms, optimized data movement, and more. By delivering a full-featured suite of capabilities, the OEM can rapidly bring embedded data reduction to market without for example having to redesign their file system or block map systems to complete the dedupe workflow.

The other attribute that accelerates time to market is simplicity. Despite being full featured, the ECOsystem for OEMs is accesses via an lightweight object-like API that OEM developers have told us is extremely simple to work with.

3) The solution has the flexibility to support specific use cases.
The requirements and functional expectations for implementing data reduction differ from device to device, and any embedded solution needs to have the adaptability to serve different applications. For example content-aware algorithms may not be a meaningful solution in a block array where data structures are completely opaque. Or CPU and memory constraints on a given device may require the use of a lighter weight dedupe workflow, and that shouldn’t force major architectural rework of the solution.

The ECOsystem for OEMs has been designed to support 6 embedded use cases: Servers, block arrays, NAS, object stores, cloud storage, and backup targets. Some of the differences between these tiers manifest themselves as implementation best practices, but there are also clear functional decision-points that allow an OEM to implement the right solution for the job. Importantly though, all of these tiers are compatible in key respects, allowing cross-platform manageability, and giving rise to end-to-end features such as optimized data movement.

4) The solution has high performance, while working within resource constraints.
Performance overhead is an often discussed problem associated with dedupe solutions. We’ve learned to solve these problems through 2 years of empirical experience, and believe we have the fastest, lowest-overhead dedupe solution. Moreover, ECOsystem for OEMs obeys hard constraints of the host platform in terms of CPU and memory usage.

There are a couple main points where dedupe can impose performance penalties:
A) During write, the chunking and lookup process takes time. Like other solutions, Ocarina does this in memory to reduce that penalty. You do have to be careful to understand for a given chunk size how much unique data that 1GB of dictionary can address, and given a constrained memory, how far can the solution go before forcing on-disk (=slowww) lookups. ECOsystem for OEMs utilizes the industry’s most memory efficient lookup design to make the most of existing resources.
B) During read, the reads from disk for any de-duplicated volume are more random than typical disk IO. Ocarina is also able to mitigate this impact through content-aware read-ahead caching that anticipates the next chunks that will be read from disk.

5) The solution supports next-generation features that customers want and competitors don’t have.
Ocarina has spent a lot of time in the market talking to end-customers about what they want in a dedupe solution. In addition to things like “shrink well”, a couple things keep coming out. One is to keep data in shrunken form as it moves around in workflow operations such as replication, backup, tiering, etc. We call this end-to-end optimization (or E2EO) and it expands the value of dedupe by improving backup processes, reducing LAN bandwidth, and reducing the CPU overhead that occurs when data is repeatedly rehydrated and re-deduped for no reason. For those OEMs who carry a broad catalog of server, storage, and backup solutions, there are huge benefits in being able to deliver this end-to-end value to customers who have adopted that OEM across their entire IT infrastructure.

The other key feature is the ability to support structured (eg database) and semi-structured (eg VM) applications. The files in these applications are almost always >95% static and <5% active. But because they are active, traditional dedupe solutions can create a tremendous IO overhead as dedupe operations battle against a steady stream of changes. Instead of getting in the way, Ocarina has invented a way to dedupe the inactive portions of these files, while imposing no performance overhead on active IO. Like our dedupe chunking process, this is one of several content-aware features that Ocarina delivers in the ECOsystem for OEMs.

These advanced dedupe features — which bring data reduction benefits to new applications and workflows — allow storage OEMs and ISVs to deliver more value to customers, to do it faster, and differentiate their product over competitive offerings that have first-generation dedupe capabilities.

Some of the discussion seen in articles around the Web (including Storwize) has focused on the speed of compression and deduplication – with concerns around how this impacts primary storage performance. Completely valid issues, as one should not have to substitute performance in exchange for features. From our company launch, we have focused on achieving the best of both worlds – with primary data storage optimization, and high performance, both with our own dedicated devices and those from our OEM partners.

For customers who want choice, Ocarina has you covered. Ocarina has both fast in-band deduplication and advanced compression options. You can either run fast in-band deduplication, with sub-millisecond latency, or you can choose deep content-aware compression, which takes longer, of course, but also gets results that simple deduplication can’t hit. Or… you can do both! Stop the presses!

Ocarina’s deduplication is fast – you can get deduplication results immediately on every file that passes through our systems, and then come back to do a post-processing run that gets advanced results later, at an off-peak time, should you choose. Also, the post-processing engine is driven by policies which you set, letting you compress only files that meet criteria you choose – for example, size, age or type.

Thanks to Ocarina winning a number of high profile deals at large customers where deduplication alone is not enough, Ocarina has become associated with heavy compression – but we do dedupe as well, and we’re quite good at it! If you look at results we recently got on a corporate data set, we were able to shrink that data set by 92% overall, with 60% coming from deduplication and 32% more coming from files that were compressed in the post-process.

At Ocarina, we believe deduplication will soon become an embedded system feature, and a commodity. It is possible to do in-band deduplication, with very little latency, and minimal CPU resource demands. Dedupe will become a storage fundamental, and the pricepoint for customers to gain dedupe will trend towards zero (where NetApp is today with their A-SIS offering).

Companies like Permabit will have to win quite a few OEM deals at these kinds of end user prices, minus an OEM discount, to be profitable – but advanced features, including dedupe-aware data movement and advanced compression, will be value-add features customers will pay more for. Unlike dedupe, those features won’t provide big value for every customer, but they will apply as important benefits providing value to a significant percentage, and unlike dedupe, advanced content-aware compression is not going to be a commodity given away for free in every system.

Ocarina is in a good position with our technology, and both customers and OEMs should evaluate not just the technology of a dedupe provider, but their ability to financially survive as well. Once a company has reduced your data, and locked it away in their format, the last thing you want is for that company to go out of business, and compress your chances of getting it back. A company that only has dedupe on the table is going to be priced out of the market by 2012, even if it is successful today selling dedupe.

When something sells to end users for free, you can’t make up the profit margin by selling in volume. In Ocarina’s case, we can be extremely aggressive on price for dedupe, because we bring more to the table and have something else to sell, and every OEM deal we win creates a platform ready for future upgrades. As Wikibon wrote yesterday, “Ocarina provides the highest levels of compression by using the optimum compression techniques.” We are the best in the world at this stuff and we will continue to stay ahead of the competition. We also have a well-rounded business that won’t see us get deduped. So if you are looking for an advanced solution that does much more than dedupe, Ocarina is the answer.

Deduplication and other data reduction technologies offer the opportunity for increased data protection at a reduced cost.

When you hit a threshold of 50% or greater overall data reduction, it lowers the cost of performing a a full mirror of data. For example, if you have 100 terabytes of data, mirroring today would require 200 terabytes of disk space. But if the data were reduced by 75%, you could store the same data in 25 terabytes, and full mirroring would require on 50 terabytes of disk - a significant savings. In other words, it is possible to fully protect your storage, including full mirroring, with less disk than it takes to store the unprotected data today.

While this is true with any amount of space savings, when that level of savings goes beyond 50% (or a dedupe ratio of 2:1 or better), the storage cost of full mirroring is zero.

There are a number of options that your deduplication provider should be pursuing to compliment your data protection strategy. For example, a vendor should be able to allow you do two key things with your dedupe configuration:

1. Allow a minimum level of duplicate blocks to accumulate prior to starting deduplication. For example, you could allow data to be written twice, and then deduplicate all subsequent occurances. So long as the dedupe solution is aware of both original occurances, you have a form of mirroring without needing to do full mirroring at the disk level. Call this duplicate mirroring.
2. Set a threshold for a maximum number of duplicates. To set a maximum level of exposure on the loss of physical disk, you should be able to ask that once you have found ‘n’ instances of the block, to start over. You could set that number at 8, 32, 128 or whatever frequency makes business sense to you, and therefore, the potential loss of the sector on which the duplicate is stored would only affect a certain number of files.

These two examples are not necessarily a complete answer in and of themselves, but they do provide guidance for tools you can work with as part of an overall data protection strategy. As deduplication becomes a storage fundamental, and is in place across multiple tiers of storage and on multiple products in your data center, understanding the impact of dedupe on your data protection strategy will be key and your dedupe vendor should be providing you the right tools to manage that.

This report gives a solid overview of dedupe and how it relates to the overall storage market, its very detailed and I would encourage anyone interested in dedupe to download it and give it a read. Cheers!

The most interesting trends from our perspective were:

1) The growth in unstructured data continues to outpace structured data,

2) Thanks largely to widespread media-enabled cell phone use, 70% of data will be user-generated

3) This one really jumps out: 35% more digital information is created today than the capacity exists to store it. This number will jump to over 60% over the next several years.

So what the IDC numbers mean is that platter density can’t keep pace with the market. Although John Toigo (http://www.drunkendata.com/?p=2872) refutes that by arguing platter densities have kept ahead of IDC’s expectations, I expect the truth to be somewhere in the middle. Either way, the ability to find more disk and datacenter to store the deluge is going to present more challenges for Sysadmins, who are already up to their necks in data.

Good news for vendors who supply storage hardware. No surprise it’s EMC who sponsors this study!

Administrative issues aside, it raises a question of IT sustainability. The IT budget in a typical Fortune 1000 enterprise needs to track as a percentage of overall capex and opex. When IT in a typical enterprise must grow as a % of overall expenses, that can’t continue; IT has to consider itself non-scalable. With the scenario described in the report, IT organizations are going to encounter this exact scenario. Their storage budget will grow as a percentage of opex and capex, and that can’t continue indefinitely.

So the market has two choices:

1) Throw away data.

2) Greatly improve utilization, through technologies like thin provisioning and data reduction.

History has already shown that throwing away is not going to happen because the IT staff managing it doesn’t own the data, and they have neither the tools nor the buy-in to begin expiring the data of their users. So primary storage optimization is getting a ton of attention. We anticipate data reduction features rapidly moving from “should have” to a “must have” in storage solutions globally, and enterprises globally are being encouraged to start evaluating reduction technology if they want to keep storage budgets in check. In fact we’re starting to see that sentiment in F1000 polling by TheInfoPro (www.theinfopro.com): Their two latest survey waves are showing Online Data Reduction as ranked 2 of 21 in their Storage Networking Technologies Heat Index.

In a perfect world hardware would be free and wouldn’t generate heat, users would diligently delete files, the IT staff would triple and we’d all win the lottery, which would allow you to safely store the 1.2 Zettabytes of data that’s crossing your LAN in 2010! But the boss says we shouldn’t count on those things, so we’re working hard on end-to-end solutions for a more intelligent way to deal with the data explosion.