Backup is a really interesting workflow, one that customers consistently highlight as an area with recurring headaches. Some of the issues we’ve heard:

• The cost and complexity of backup software and its licensing,
• The administrative hassles of managing tape,
• Major difficulty identifying performance bottlenecks,
• Managing the day-to-day requests for restore operations,
• Business problems like making sure the management’s expectations for retention and recoverability are in line with the protection solution they’ve budgeted for.

One of the most time consuming activities for the administrator is restore operations, especially if it involves transporting tapes back from a vaulting service. The response time to a user who “accidentally blew away his file” or a request to restore an x-employee’s inbox, is substantially improved if that backup is staged on disk. That’s not to say disk is faster than tape; it certainly can be, but you can also get a heck of a lot of throughput from a stack of LTO-5 tape drives! It’s when the RTO includes the time to drive a truck, find a cartridge, spool, shoe-shine and stream…that’s when the responsiveness to the end-user falls apart.

So we want to store as many backup images on disk as possible, but the capex and opex of a JBOD are hardly in line with the costs for tape. But it turns out that dedupe algorithms are a match made in heaven for backup workflows. If we can shrink your data 15-20x and thus capex/opex, then the staging of backup images can go from say 1 week to say 15 weeks, and the likelihood you’ll get a restore request for something over 15 weeks old is pretty low. Yeah! No restores from tape!

We built the DR4000 just for the needs of the SMB to mid-market business, where backup sets are 1-5TB, and customers want simple streamlined value that plugs-and-plays in existing backup environments. Generally this is a set-it-and-forget-it purchase for customers, with most of the administration of backups and restores happening through the backup software.

Here are some of the specs:

• 3 models; 2.7, 5.4, and 9TB usable capacity (before dedupe)
• Special in-band dedupe and compression algorithms coming from the Ocarina team
• Optimized replication: Only deduped data goes over the wire
• 2u chassis, RAID6 protection, 1G and 10G connectivity, GUI and CLI administration, NFS/CIFS/OST
• Initial certification with Commvault and Symantec (NBU and BE), with others coming soon
• All inclusive licensing so you get all the future enhancements for free

The DR4000 is one of what will be many investments in how we improve the protection solutions available to customers. In the future we’ll be adding more to this family, but where it gets interesting is when we apply Fluid Data principles to the roadmap, including ideas like more seamless integration with EqualLogic and Compellent, as well as Cloud services. More to come this year!

What you are seeing here is the transformation of Dell from a storage reseller to a storage company. This means that we have become a leading creator of new storage technology. The companies we’ve purchased not only have great products, but have great development teams, teams that are consistently pushing forward the frontiers of storage technology. People have been asking about what Dell plans to do with all these companies. Each one contributes not only an industry-leading product, but also some key technology that fits into Dell’s Fluid Data vision. Our strategy is to dramatically improve the way storage supports your business objectives at the same time that we dramatically improve the economics of buying and deploying storage.

The foundation of the Dell Fluid Data Architecture started three years ago with the wildly successful acquisition and integration of SAN provider EqualLogic. At the time, people said Dell was taking a risk on the unproven iSCSI market. Today, Dell Equallogic is the market leader in iSCSI, the fastest growing segment in storage over the last 5 years. Exanet adds a scalable, fault-tolerant file system that we will roll out across all our product lines, starting with the launch of the PowerVault NX3500. Ocarina will give us consistent and compatible dedupe across every product – which means that you get the benefits of data reduction not only on disk, but in every storage workflow that moves data. Compellent brings us an enterprise-scale storage platform, as well as industry-leading technology for dynamic storage tiering. Compellent also contributes the term “Fluid Data” – they’ve been using it to describe some of the breakthrough elements of their array, and it turns out to be a very good term to describe what we are doing with the whole product line. Your business is dynamic, your data needs to be fluid – it needs to flow to the right place at the right time to get the job done.

Dynamic storage tiering is a great example of Fluid Data. Today, many companies put all their data on fast, expensive storage, because they are not sure which data is active, and which data is not that active. Better to be on the safe side. But being on the safe side is expensive and inefficient. By collecting statistics on usage, Fluid Data can determine in real time where data needs to be. If you can deploy a mix of fast storage and less inexpensive disk, knowing that dynamic storage tiering will make sure your data is in the right place at the right time, then you can not only save money but improve performance as well. This is just one example of the Fluid Data Architecture in action.

One of our most gratifying proof points is that customers like you see the incredible value of our best-in-classes offerings. We’re helping many customers worldwide, including Websense, Infinity Pharmaceuticals, South West Community Care Access Centre, Carnival Cruise Lines, AccuWeather and many others.

Our storage vision has always been – and will continue to be – making storage more capable and more affordable for our customers. This vision guides our development strategy and every acquisition we make. It is truly a joy sharing how all of the pieces are coming together to provide a consistent and compatible experience – directly impacting how you manage your information and grow your business. I look forward to telling you more about what we are doing. In my next blog post, I will go through the key tenets that define the Dell Fluid Data Architecture.

See what others are saying about the Dell vision at Computerworld and SearchStorage.com.

But the general understanding of data loss mechanisms seems to be limited to anecdotal information and participants can’t say for sure whether component failure is a bigger culprit than say human error or media failure. David Rosenthal from Stanford emphasized rightly that no one has developed a realistic failure model for preservation archival that takes into account the complete picture, including that failures are correlated (for example more human error happens when the system needs maintenance). Thus the few vendor product characterizations out there have grossly exaggerated the solutions’ reliability in practice.

One consistent thread in the conference was the request for beyond-enterprise-class reliability features, Yet many in the room (DOD users aside ) have no money to buy enterprise-class storage and have little more than white box PC’s with RAID5 SATA disks stacked in a closet. Some are even storing archival collections on home-made object-storage and WAN replication solutions. Probably not a good idea without a solid DR strategy (fortunately traditional backup is not a requirement when data is completely fixed). Meanwhile, well funded institutions like LoC and Stanford maintain sophisticated storage strategies, striking a balance between goals of availability and preservation.

Dell was there to provide some guidance and vision on how dedupe and compression should be considered as an legitimate element in preservation archival, especially given most archives have budget issues that limit what they can save in the first place. It’s not a surprise that long term preservationists have a natural anxiety against anything proprietary that renders the bits into something that can’t be natively read by VI. There’s a subjective risk coefficient associated with “How can I make sure the files are readable in 10 years when the product is obsolete?” or even “How can I be sure that Dell will even be around in 80 years?”

The bottom line is now clear; that data reduction is now going to be a standard storage feature, available across entire families of products. Although there are proprietary mechanisms in all of these data-reduction solutions, the threat of the “100-year problem” can be mitigated through self-describing compression formats and software-based readers that can be stored alongside the data and flexibly deployed for future readback. In terms of compression, it’s worth noting that users crossed that bridge a long time ago when they archived JPEG images and MPEG1 video. No one should assume that 2015′s version of VLC can play that video, and preservationists are acutely aware of these format longevity issues.

The Library’s Lead Storage Engineer Thomas Youkel emphasized that a big part of preservation archival is planning for both format migration (eg re-encoding old video to h.264), media migration (moving it from 3/4″ tape to LTO4), and platform migration (replacing an old fibrechannel SAN with object storage) to take advantages of enterprise support and improvements in performance/capacity/density. With each migration — roughly at 5 year intervals and sometimes taking multiple years to accomplish — the archive owner has the opportunity to add technologies with promising value or remove any technologies that didn’t deliver. So in the end, the 50 or 100-year preservation concern may not be relevant in the context of IT-oriented decisions. As long as you can trust Dell to be here in 5 years (and I’ll bet my boss’s paycheck on that), then the storage platform is taken care of.

Check out the Library’s initiatives at www.digitalpreservation.gov

The acquisition of Ocarina by Dell has now closed, and we can now say some things about where we are headed.  Mergers and acquisitions can often be a time of confusion, as the new combined company figures out what the new plan will be.

I’d like to use this first blog after the closing of the deal to clear up some of that confusion.    Some of our competitors have tried to take advantage of the period between the announcement of the deal and the closing (a time during which neither company can make forward-looking statements about joint operations) to sow uncertainty and doubt amongst some of Ocarina’s most-valued customers:  our OEM and resale partners.

Now that the deal has closed, I can say that Dell is actually very interested and committed to pursuing OEM partnerships for the Ocarina technology with other storage vendors.

One competitor in particular tried to imply that our OEM’s would have to drop Ocarina and go looking for a new dedupe technology.

This is simply not true.  While each partnership is a unique case, in general, the OEMs that want to continue doing business with us will be able to. What’s more, this is now a stronger solution, both technically and on the business front.  Dell plans to invest in growing the Ocarina engineering team – and provide a richer roadmap and have a faster cadence.    Clearly, onsite support will now be available worldwide for those partnerships that involve reselling an Ocarina-based dedupe appliance. Software support will eventually be available in many more languages, also around the clock. In addition to those obvious technical benefits, some of the risk of doing business with a startup is now also resolved.

Finally, I think that Dell sees value in an ecosystem of compatible and complimentary dedupe products. It’s great, for example, that Ocarina dedupe can work together with Commvault and Symantec dedupe. We think it is good for customers to see that compatible ecosystem grow. What is good for customers is also good for our OEM’s – by participating in an open ecosystem that gives customers not only the best-of-breed technology, but also a rich set of choices for deployment, they are strengthening the solution they offer and the story they have to tell.

What was even more impressive about this survey was the respondents themselves. While it’s easy to understand for the largest companies with their heavy data sets that scalability is a major strain, two thirds of these respondents reported having less than 50 terabytes of storage, meaning these challenges have now hit mid-size businesses as well. The <50TB crowd is going to be more risk averse, and reducing their storage from 3 arrays to 1 array isn’t necessarily the “killer” value proposition. The value proposition of compression and dedupe technologies includes things like reduced backup windows, which can only be obtained by a solution that supports end-to-end benefits, proves to customers that it’s trustworthy and doesn’t carry additional administrative burden.

Also intriguing was the comment by many respondents that the growth of storage was in transactional applications (database and e-mail) rather than unstructured data. Many of Ocarina’s clients in multiple market segments are seeing dramatic growth in unstructured data, but in this business tier, more traditional applications are adding to the data glut. Also, there is little doubt that in a short amount of time VM applications will appear as part of that category. This is also in contrast to the “Petabyte vertical markets” which are chock full of unstructured file data, and a good lesson for OEMs to take away; any embedded dedupe implementation needs to deliver results in structured-data applications, and that’s a non-trivial requirement. Common dedupe solutions in the market today would create noticable performance impact on these apps, which is why Microsoft removed single-instancing as a native feature from Exchange. A dedupe solution in these applications needs to include workflow and application awareness, for example compressing and deduping only those portions of the database (or VMDK) that are inactive. Ocarina calls this Heat Index Management, and it’s a feature included in our ECOsystem for OEMs.

As industry observers have seen in the last several years, backup has been the killer driver for deduplication – the first wave, if you will, because the ROI of D2D backup is tremendous with this feature in place. But in the survey there is strong adoption (>50% of deployments) of dedupe for archival and other non-backup apps. Of course this isn’t unexpected to Ocarina, because we’ve helped drive this shift towards compression on primary storage and the entire data lifecycle.

As data growth continues in all markets, we believe strongly in the need to compress and optimize across the datacenter, from primary storage to backup and archival. InformationWeek has done a fantastic job demonstrating real customer demand for these technologies, which we expect will increase. If you haven’t walked through the presentation, make sure you do at http://www.informationweek.com.

When I heard the news yesterday that Dell would be acquiring Ocarina Networks, my first reaction was pride. I couldn’t help but give myself an inner “high five” for having recognized what a great company Ocarina is and would become. My second was to congratulate Dell on their smart choice. I was lucky enough to have worked with the people at Ocarina, and now a whole new group of folks will have this opportunity.

The people at Ocarina Networks are not run-of-the-mill. They are not even above average. They are extraordinary. Anyone who wants to understand what makes a technology company excel would do well to study them. As I wrote in my final, farewell post, Murli Thirumale, CEO of Ocarina is about as far from the image of the typical start-up CEO as you can get. He is soft spoken, thoughtful, and a good listener.

He built his company around a well thought out business philosophy. He first identifies a problem or need, and then designs a product in response to it. In this case, the problem he identified was the proliferation of data worldwide. Rather than go for the obvious, he brought together a team to research the problem and solve it in a new way. This approach is rare. Most of the time, you have start-ups that try to launch based on a technology they themselves developed–in other words, their own pet project. It takes discipline to go about things the way that Murli does.

Murli also showed himself to be a leader in the true sense. Rather than feed his own ego, he chose to surround himself with giants of the storage industry, and he allowed them to get the job done the way they saw fit. Carter George, VP Products, is one of those people. I am particularly indebted to Carter. He was truly a mentor to me. Despite the intense demands of his role at Ocarina, he always had time to answer my questions. He never treated me as anyone other than an equal. I also got to know Goutham Rao, the visionary genius who is the company’s CTO. In addition, I met Dave Withers, who is the man behind the company’s multiple partnerships with top storage companies. For the last few months of my tenure, I worked closely to ramp up the social media program with Mike Davis, director of marketing. Dell is extremely fortunate. They are acquiring technology, but to me the real gift is that they get to work with these remarkable people.

As the headline says, this to me seems like the beginning for Ocarina. It is a bold plan to take the vision of end-to-end dedupe and make it real. This is the next step for the storage industry–and a crucial one if storage costs are to be kept in line, and infrastructure is to keep up with the demands of the real world. We are living in a time when data growth is spiraling upward at rates that no one could’ve imagined even a decade ago. This is the time for a cohesive, meaningful response to this reality. I couldn’t imagine a better outcome for Ocarina, Dell, and the industry as a whole. I am honored to have been a small part of it.

For our people, it is a great outcome.  Good people build great technology.  Great people build products that get used over and over again by a large number of customers.  Getting our world-beating dedupe solutions into the hands of a large number of customers rapidly is what we are all about.

We have been working with Dell for over a year.  The people we have met all the way from engineers up to Michael Dell are intellectually curious, direct, driven to succeed,  talented and fun to work with.  In fact, they are a lot like us at Ocarina! Culture alignment?  You bet baby! Finally, even with its success, Dell’s storage business is relatively new and their willingness to fund it (they bought us didn’t they?), long-term expansion plans and focus on talent acquisition ensures a great opportunity for the talented team we bring across all functions to Dell.

Primary dedupe leadership by the year 2012.  Now coming to you at rocketship speed powered by Ocarina AND Dell.  Watch out storage industry!

Here are a series of requirements that we’ve consistently heard from OEMs in our discussions, and here’s how Ocarina addresses those in the ECOsystem for OEMs.

1) The solution shrinks data well.
Ok, so this is obvious. Dedupe is supposed to shrink data, and that improves system utilization and reduces costs. Doing this well is about algorithms, and Ocarina has been on top in the algorithm game since we started shipping our primary storage optimization products almost 2 years ago. We’ve shown in some deployments that we can deliver 80% savings when traditional block dedupe delivers no more than 30%. We believe in fact that we’re the only storage technology vendor to actually employ algorithm PhD’s whose sole job is to invent better algorithms…which is something that has proven really useful in specialized markets where a few unique content types dominate the terabytes.

We are the only provider today that deploys dedupe and compression concurrently. Our dedupe algorithm is a content-aware, variable-block, sliding-window approach. ‘Content-aware’ is an overused term for sure, but here it reflects for example that we’ll recognize monolithic data structures in a stream like a JPEG blob, and we know that slicing that JPEG into 8KB chunks for dedupe delivers absolutely no benefit, and thus is a complete waste of time, CPU, and memory. We’ll treat that JPEG (and other data types like it) as a contiguous chunk, which makes our dedupe namespace extremely fast and efficient.

2) The solution minimizes time to market.
Most OEM vendors are under considerable time pressure to bring dedupe to market, either as a competitive response, or because their customers are demanding it. But the OEM’s dev and test engineering resources are always limited. With that in mind we made a point of developing a full featured library. That means not only do we slice the data and do hash lookups (as with the Permabit product), but we also do the dedupe, compression, on-disk data management, metrics and reporting, throttling mechanisms, optimized data movement, and more. By delivering a full-featured suite of capabilities, the OEM can rapidly bring embedded data reduction to market without for example having to redesign their file system or block map systems to complete the dedupe workflow.

The other attribute that accelerates time to market is simplicity. Despite being full featured, the ECOsystem for OEMs is accesses via an lightweight object-like API that OEM developers have told us is extremely simple to work with.

3) The solution has the flexibility to support specific use cases.
The requirements and functional expectations for implementing data reduction differ from device to device, and any embedded solution needs to have the adaptability to serve different applications. For example content-aware algorithms may not be a meaningful solution in a block array where data structures are completely opaque. Or CPU and memory constraints on a given device may require the use of a lighter weight dedupe workflow, and that shouldn’t force major architectural rework of the solution.

The ECOsystem for OEMs has been designed to support 6 embedded use cases: Servers, block arrays, NAS, object stores, cloud storage, and backup targets. Some of the differences between these tiers manifest themselves as implementation best practices, but there are also clear functional decision-points that allow an OEM to implement the right solution for the job. Importantly though, all of these tiers are compatible in key respects, allowing cross-platform manageability, and giving rise to end-to-end features such as optimized data movement.

4) The solution has high performance, while working within resource constraints.
Performance overhead is an often discussed problem associated with dedupe solutions. We’ve learned to solve these problems through 2 years of empirical experience, and believe we have the fastest, lowest-overhead dedupe solution. Moreover, ECOsystem for OEMs obeys hard constraints of the host platform in terms of CPU and memory usage.

There are a couple main points where dedupe can impose performance penalties:
A) During write, the chunking and lookup process takes time. Like other solutions, Ocarina does this in memory to reduce that penalty. You do have to be careful to understand for a given chunk size how much unique data that 1GB of dictionary can address, and given a constrained memory, how far can the solution go before forcing on-disk (=slowww) lookups. ECOsystem for OEMs utilizes the industry’s most memory efficient lookup design to make the most of existing resources.
B) During read, the reads from disk for any de-duplicated volume are more random than typical disk IO. Ocarina is also able to mitigate this impact through content-aware read-ahead caching that anticipates the next chunks that will be read from disk.

5) The solution supports next-generation features that customers want and competitors don’t have.
Ocarina has spent a lot of time in the market talking to end-customers about what they want in a dedupe solution. In addition to things like “shrink well”, a couple things keep coming out. One is to keep data in shrunken form as it moves around in workflow operations such as replication, backup, tiering, etc. We call this end-to-end optimization (or E2EO) and it expands the value of dedupe by improving backup processes, reducing LAN bandwidth, and reducing the CPU overhead that occurs when data is repeatedly rehydrated and re-deduped for no reason. For those OEMs who carry a broad catalog of server, storage, and backup solutions, there are huge benefits in being able to deliver this end-to-end value to customers who have adopted that OEM across their entire IT infrastructure.

The other key feature is the ability to support structured (eg database) and semi-structured (eg VM) applications. The files in these applications are almost always >95% static and <5% active. But because they are active, traditional dedupe solutions can create a tremendous IO overhead as dedupe operations battle against a steady stream of changes. Instead of getting in the way, Ocarina has invented a way to dedupe the inactive portions of these files, while imposing no performance overhead on active IO. Like our dedupe chunking process, this is one of several content-aware features that Ocarina delivers in the ECOsystem for OEMs.

These advanced dedupe features — which bring data reduction benefits to new applications and workflows — allow storage OEMs and ISVs to deliver more value to customers, to do it faster, and differentiate their product over competitive offerings that have first-generation dedupe capabilities.

Some of the discussion seen in articles around the Web (including Storwize) has focused on the speed of compression and deduplication – with concerns around how this impacts primary storage performance. Completely valid issues, as one should not have to substitute performance in exchange for features. From our company launch, we have focused on achieving the best of both worlds – with primary data storage optimization, and high performance, both with our own dedicated devices and those from our OEM partners.

For customers who want choice, Ocarina has you covered. Ocarina has both fast in-band deduplication and advanced compression options. You can either run fast in-band deduplication, with sub-millisecond latency, or you can choose deep content-aware compression, which takes longer, of course, but also gets results that simple deduplication can’t hit. Or… you can do both! Stop the presses!

Ocarina’s deduplication is fast – you can get deduplication results immediately on every file that passes through our systems, and then come back to do a post-processing run that gets advanced results later, at an off-peak time, should you choose. Also, the post-processing engine is driven by policies which you set, letting you compress only files that meet criteria you choose – for example, size, age or type.

Thanks to Ocarina winning a number of high profile deals at large customers where deduplication alone is not enough, Ocarina has become associated with heavy compression – but we do dedupe as well, and we’re quite good at it! If you look at results we recently got on a corporate data set, we were able to shrink that data set by 92% overall, with 60% coming from deduplication and 32% more coming from files that were compressed in the post-process.

At Ocarina, we believe deduplication will soon become an embedded system feature, and a commodity. It is possible to do in-band deduplication, with very little latency, and minimal CPU resource demands. Dedupe will become a storage fundamental, and the pricepoint for customers to gain dedupe will trend towards zero (where NetApp is today with their A-SIS offering).

Companies like Permabit will have to win quite a few OEM deals at these kinds of end user prices, minus an OEM discount, to be profitable – but advanced features, including dedupe-aware data movement and advanced compression, will be value-add features customers will pay more for. Unlike dedupe, those features won’t provide big value for every customer, but they will apply as important benefits providing value to a significant percentage, and unlike dedupe, advanced content-aware compression is not going to be a commodity given away for free in every system.

Ocarina is in a good position with our technology, and both customers and OEMs should evaluate not just the technology of a dedupe provider, but their ability to financially survive as well. Once a company has reduced your data, and locked it away in their format, the last thing you want is for that company to go out of business, and compress your chances of getting it back. A company that only has dedupe on the table is going to be priced out of the market by 2012, even if it is successful today selling dedupe.

When something sells to end users for free, you can’t make up the profit margin by selling in volume. In Ocarina’s case, we can be extremely aggressive on price for dedupe, because we bring more to the table and have something else to sell, and every OEM deal we win creates a platform ready for future upgrades. As Wikibon wrote yesterday, “Ocarina provides the highest levels of compression by using the optimum compression techniques.” We are the best in the world at this stuff and we will continue to stay ahead of the competition. We also have a well-rounded business that won’t see us get deduped. So if you are looking for an advanced solution that does much more than dedupe, Ocarina is the answer.