Online Storage Optimization

As many people know, Ocarina Networks has been living up to its name lately. It really is becoming a “network”-oriented company, inking partnerships with just about all of the top storage vendors–HP, BlueArc, Isilon, and so on and so forth. This is great news for storage customers, who can now depend on the very best in data reduction, slashing storage costs.

For those who want to get a quick and entertaining hit on how one of these partnerships works–this one with BlueArc, might we suggest this new animated demo on the Ocarina site? This demo offers a case study in how a world-class CGI animation studio, Rainmaker Entertainment, deployed BlueArc storage with Ocarina to achieve astounding compression results. (For more on this, you might also want to take a look at our Q&A with Shmuel Shottan, CTO of BlueArc from last February.)

And for more on how Ocarina is joining forces with the top storage vendors to help media and entertainment companies maximize storage capacity, check out these recent news stories:

Beth Pariseau, SearchStorage - Ocarina deduplication and Isilon clustered NAS help visual effects studio archive images, cut costs

Debra Kaufman, Studio Daily - VFX Companies Lower Storage Costs with Ocarina

Bryant Frazer, Studio Daily - Q&A: Carter George, VP of Products, Ocarina Networks
Downsizing Storage Requirements for Post-Production

Happy Friday everyone!

A few of the Ocarina crew recently returned from Siggraph2009, the 36th International Conference and Exhibition on Computer Graphics and Interactive Techniques. Held in New Orleans the week of August 3-7, the event drew participants from around the world. We were newbies to the event, and so decided get some perspective from an industry veteran, who has been attending Siggraph every year for the past decade and a half.

Michael Zachary Huber is an animator and educator who has worked with the top studios, including director James Cameron, Digital Domain, and Electronic Arts (EA). He’s an assistant professor at Cogswell Polytechnical College an animation and engineering school in Sunnyvale. Over the years he’s witnessed peaks and valleys when it comes to Hollywood’s love affair with visual effects.

“In the 1990s they were a novelty,” he said. “It was similar to the Internet, which came into its own later in that same decade. People were just on fire!”

The early 1990s were truly the heyday of Siggraph, he recalled.  It wasn’t unusual to see stars like Danny DeVito and Arnold Schwarzenegger in attendance, and there was tremendous “buzz” in Hollywood. Yet some studios were burned by spending millions of dollars for lavish effects for movies that flopped.

He compares the new animation-driven effects such as CGI to the craze around 3-D movies in the 1950s. That type of special effect didn’t make movies better, just more novel.

“In some ways visual effects are the same thing. They are a tool, not an end unto themselves. The movie still has to be good for audiences to respond,” said Huber.

Nowadays directors and studios are getting smarter about where visual effects need to be used and where they don’t, he said.

Siggraph itself has been something of a barometer as to how the animation and effects side of the industry is faring. And this year, the attendance was far lower than in recent years, perhaps by as much as 25-50% in his estimation.

Huber’s interpretation–it’s not that the recession has meant that the industry is in real trouble, simply that this is a year in which studios are more cautious, but are still very much investing in the coming year. Said Huber, “I firmly believe there’s going to be a nice rebound.”

An exciting and fun part of the conference he said, is the Computer Animation Festival, where participants from around the world screen their latest work. Check out this extremely cool vid showing some of the work:

Huber himself has a short animation film, a co-production with Cogswell that he plans to show at next year’s event. Called “The Offering,” it’s a story that draws from Hindu legend yet includes elements from everything from Marvel Comics to Bollywood. It was made at the school, with students playing a large role in its creation. (The poster for the movie is shown at the top of this page.)

So, what about the geeky side of Siggraph?

“The conference definitely gets a steady stream of techno fans–people who are interested in the technology and want to come for the white papers,” he reported, adding that the need for efficient storage is something that many animation houses are recognizing.

“Looking at the first Transformers movie, which was made four or five years ago, Industrial Light and Magic (ILM), which did most of the effects, used about 30 TB of space for all of the files they needed during the production process. Compare that with the latest Transformers movie, which took up closer to 150-250 TB of space,” said Huber.

Even his own short film took up three terabytes of space to make, he said. So even small institutions that have limited budgets could benefit from some kind of compression or optimization technology.

Indeed, he said, the computer animation industry is perhaps the art form most closely tied to technology. This, he said, is another reason that this year’s Siggraph conference was less well attended–the tech industry is smarting from the effects of the recession.

Yet it’s also the pace of technological innovation that drives visual effects/animation studios to continually improve what they’re able to achieve. Effects such as 3-D animation are getting more complex, he said, especially now that studios have moved from 8-bit to 16- or 32-bit technology. One exciting new innovation is OpenEXR, a high dynamic-range (HDR) image file format developed by ILM. This allows studios to ingrain details in visual effects like never before, according to Huber.

“You’re not going to get the true richness unless you get the file formats like that,” he said.

However, he added that such new file formats are demanding and require a lot of space. This is no doubt why so many post-production and animation studios are looking for ways to optimize files in order to save disk space. As it happens, Ocarina stands alone in that it has algorithms that are designed specifically for over 100 file types–OpenEXR included. As many studios are already discovering, Ocarina is their ticket to space savings of as much as 80%.

Overall, Huber sees a bright future for his industry.

“We see films today that seem amazingly more complex and rich.  In ten years, all of that will be topped by what is coming.”

Well, I for one can’t wait for those coming attractions.

Today was a great day for Ocarina. Not only are we attending SIGGRAPH2009, the premiere event for graphics artists and movie animators, but we also announced two major customers in that realm, Rainmaker Entertainment and Zoic Studios. Both are finding that they’re able to achieve 80% reductions in data as a result of deploying our content-aware technology. Truly something to sing in the rain about.

Below find the entire press release:

Media and Entertainment Giants Turn to Ocarina for Online Storage Optimization

Rainmaker, Zoic and Other Leading Production Studios Achieve up to 80% Reduction in Storage Costs

SAN JOSE, CA–(Marketwire - August 5, 2009) - Ocarina Networks today announced that Rainmaker Entertainment and Zoic Studios are the latest customers in the CG animation, post-production and visual effects (VFX) industry to implement its ECOsystem solution to achieve up to 80% reductions in their primary storage footprint.

The media and entertainment industry is experiencing up to 50% data growth per year, driven by the adoption of all-digital HD and film workflows and the move to 3D production. At the same time, there is a growing demand for online archival storage in response to constant repurposing of media assets.

Using Ocarina, Rainmaker and Zoic join other leading media and entertainment companies that have tapped the technology to cost-effectively deduplicate and compress their multi-tier storage systems, keeping more data online longer and creating new efficiencies in artist and animator workflow. By dramatically optimizing storage, the companies also save on data center space, power, and cooling for each production or media archive.

“The uptake in interest in Ocarina in the media and entertainment space has been dramatic and a real validation of the work we put into supporting post-production data sets. In the last month, we’ve started more evaluations, benchmarks and data quality tests in studios and post-production houses than we have in the last six months. Our wins at Rainmaker and Zoic confirm that we’re solving real technology challenges in this market,” said Dave Withers, vice president of business development at Ocarina Networks.

Production data from existing Ocarina customers indicates that capital expense for new storage may drop 70% per production, even when the data is media file types that are already compressed, and reduce the overall cost of storage by as much as 80%. Ocarina’s solution combines object deduplication with an industry-specific set of compression algorithms that can quickly and losslessly shrink key post-production file types.

Rainmaker and Zoic: Animated about Storage Optimization

Industry leading computer graphics animation house Rainmaker Entertainment has achieved a 65% capacity savings by deploying Ocarina ECOsystem. With 70 terabytes of live data, growing at 25 terabytes per year, in addition to many terabytes of archival data, Rainmaker needed to keep more data online for continued use and reference but simply did not have the necessary capacity. By leveraging the Ocarina content-aware storage optimization technology, the animation house now keeps all relevant data online and accessible to artists and production crews at all times. Instead of requiring IT to get involved in data retrieval from tape, resulting in long wait times and reduced productivity, artists can now search for and access the data in a real-time, self-service capacity.

Award winning digital studio, Zoic Studios was similarly inundated with data that they needed to keep accessible for artists. After comparing Ocarina to several other solutions, Zoic ultimately selected Ocarina for its easy and seamless integration with existing storage systems, in addition to the ease-of-use for the end-user. By deploying Ocarina they were able to reduce data by up to 70%, and increase online storage from six months to two years worth of files.

“Using Ocarina has greatly simplified and consolidated our archive process — you just turn it on and it works,” said Saker Klippsten, IT manager, Zoic Studios. “Not only has it increased our nearline storage capacity, allowing us to keep everything we need online, but also drastically reduced the amount of time required to manage the archive process and aid artists in retrieving content from tape. Our artists are more productive, our IT department has more time to devote to other mission-critical tasks, and the cost savings achieved has opened up our budget to fund new IT projects.”

At least two other major movie studios have experienced similar data reduction and cost savings as a result of deploying the Ocarina ECOsystem. Validated by independent fact-checking analyst firm TechValidate, those studios report:

--  Space savings of up to 65% on online storage;
--  Support of 1 PB media archive on only 250TB of physical storage;
--  Deployment of a cost-effective nearline archive that significantly
    increases operation productivity, while reducing the net price of disk;
--  All storage optimization achieved losslessly with perfect data
    integrity.

“Using Ocarina has made it possible and cost-effective for Rainmaker to maintain an online archive,” said Dave Algar, principal data administrator, Rainmaker. “Any CG animation, post-production or FX studio, small or large, might benefit from Ocarina’s unique data reduction technology because it is specifically designed to address the growth, performance, and data availability challenges that are facing our industry.”

About Ocarina

Ocarina is a leader in online storage optimization solutions. Organizations of all sizes use Ocarina’s content-aware optimization technology to reduce their storage footprint and achieve a ten-fold capacity increase on their current storage systems. Based in San Jose, Calif., Ocarina is privately-held and financed by leading investors JAFCO Ventures, Kleiner Perkins Caufield & Byers and Highland Capital Partners. For more information, visit www.ocarinanetworks.com

Image: Web Brittanica

This week, Ocarina sent a contingent to SIGGRAPH 2009, a conference and exhibition that draws computer graphics, animation, post-production, and imaging specialists from around the globe. Lots of toys to look at, and stories to hear. We are there through the end of this week, spending time in the booths of two of our partners, Isilon Systems and BlueArc. We’re so tired we almost feel like we’re getting wall-eyed, but every time we look up we see more folks who have gotten in their cars and driven here. There are so many new talents here, it’s really a group of incredibles. As far we can tell, the blue sky is the limit to what they can achieve.

If we were ever wondering whether next-generation data reduction technologies are needed in this era of animated, 3-D movies, the stories we’re hearing at this show convinced us that without some type of data reduction we could heading for an ice age. Data growth for this industry is a clear and present danger.

This has been a great week already for BlueArc, which announced that Starz Animation used its storage for a new animated feature, “9″ from Focus Features. For those who are here, we encourage you to come tomorrow, as cinematographer John Hickson, IT Systems Architect at Starz, and the champion of the Studio Sys Admins user group is going to speak in the booth about the installation and the movie itself. *Thanks Julie Herd Goodman for that correction.*

One of the Starz folks, Terry Dale, is quoted in the news release as follows:

“During the production of Focus’ 9, it was critical that we have a storage system in place that would handle the massive demands of the three films we were animating concurrently. 9, in particular, has characters — specifically the machines and the seamstress — that generate enormous data sets during rendering,” said Terry Dale, VP, operations, for Starz Animation Toronto. “The BlueArc Titan storage system gave us the performance we needed, the reliability we required and enabled us to complete our production on 9 without disruption.”

And in case you haven’t seen it, here is the preview for this amazing looking Tim Burton extravaganza:

Star Trek is out in theaters, and the geek world is jumping for joy–not to mention having a little fun with the technology that is and always has been a mainstay of the franchise. There are lessons to be learned from comparing Trek’s capabilities to today’s reality, especially since we stack up pretty well.

ChannelWeb’s slideshow Why Star Trek Needs a VAR demonstrates that even the 23th century could use an upgrade. They point out that the Enterprise, with its 10-story building sized data center, might actually have less sophisticated storage devices than today’s data centers, which are equipped with virtualization and other efficiencies.

Might we add that the starship would also benefit from the next generation of intelligent compression and deduplication, which would no doubt please the ship’s computer no end. Surely, the data is mainly unstructured and full of already compressed files, from its ship’s log items to ballistics analysis to medical imaging files from sickbay.

Security woes are nothing new, but one would think they’d have most of them patched by the time our great-great-great-grandchildren are signing onto their Facebook pages. Apparently not. ComputerWorld’s Ira Winkler takes us where no journalist has gone before–offering a blow by blow description of security failures (and the lessons we can learn) as depicted in the movie. Warning, that section of the article is full of spoilers.

It’s somehow reassuring to see familiar corporate logos lasting into the future. As several blogs are tracking, Nokia has some serious product placement in the film, with emphasis on one particular touch screen phone moment. This issue also came up in this week’s episode of Wait, Wait Don’t Tell Me, which also got into Trekmania with its “Not My Job” quiz on the original James T. Kirk, William Shatner.

There has, of course, been a lot of chatter on Twitter about Star Trek. One example is a discussion I had with fellow storage bloggers @Storagezilla and @Stu about what would happen if they put the Star Trek Prequel together with the Star Wars Prequel. Would Spock’s logic win, or are Yoda’s Jedi powers too great even for that clever Vulcan? Hard to say.

Photo: GeekSugar

This week, many of our crew are attending NAB Show, the annual mega-convention of the National Association of Broadcasters in Las Vegas. Why do we need to be there? Well, more and more, the business of making shows and movies has become inextricably linked to the need to store immense files.

To add to the excitement, we are announcing partnerships with two major storage vendors, HDS and BlueArc today. Both are making their storage Ocarina Optimized, thus adding our groundbreaking compression and dedupe to their offerings.

Updates: Beth Pariseau has a very well written and researched SearchStorage article up that explains the partnerships in depth. Definitely worth a read!

On Wednesday, Chris Mellor at the UK Register wrote about the BlueArc-Ocarina partnership as well, with this piece.

With BlueArc, we’re announcing the Ocarina Optimizer for BlueArc. This integrates Ocarina’s data reduction technology seamlessly with the BlueArc Titan storage solutions. CGI animation leader Rainmaker Entertainment–makers of kids animation movies, most recently an animated sequel to the original, Jerry Lewis “The Nutty Professor”–finds this combined solution an ideal method of keeping files accessible during the making of an entire CGI film, which can easily run into petabytes of data.

With HDS, we’re joining the Hitachi Data Systems Technical Alliance Partner (TAP) ISV Program. In doing so, Ocarina helps HDS customers “stretch their IT storage dollars to support the phenomenal growth of unstructured data,” as Asim Zaheer, vice president of product and competitive marketing, Hitachi Data Systems put it.

Exciting times for this industry, as well as our company. We hope to see you at NAB if you’re there.

Some worthwhile articles and posts out there this past week. Here’s what is on our reading list so far:

Don’t Stop Investing in IT - Forbes profile of NetApp CIO Marina Levinson.

Spike Lee and Storage - Pete Steege’s Seagate blog post includes a video with Spike Lee and his film editor in which they discuss the advantages of digital storage in accessing deleted scenes.

Mark Hurd’s Moment - Profile of HP CEO Mark Hurd in the current edition of Fortune.

Cloud Computing Roundtable Postmortem - Christopher Kusek (@CXI) discusses last week’s TechCrunch cloud computing panel, with links to some new offerings and startups in the space. Crackpot Ideas has also weighed in on the question of whether, as one panelist put it, “Cloud is the new dotcom.”

Storage Headlines from SearchStorage - Not to send you on a link hunt, but there’s a good lineup of last week’s storage headlines from Beth Pariseau, et. al. at SearchStorage, including HDS acquisition news of note. They’ve just added a new feature–date stamping–that makes it easier to track postings.

The term “deduplication for primary storage” has become the latest industry buzz phrase. Yet, how much do any of us know about it? This week, I sat down with Shmuel Shottan, CTO of BlueArc, and learned a great deal about what makes this emerging technology such a crucial one at this time.

I was impressed by his soft-spokenness and ability to discuss these new innovations in layman’s terms. This is clearly a great year for BlueArc, which was recently awarded the Gold prize in the SearchStorage Product of the Year Award, for the Titan 3200. Our conversation is below.

Sunshine: Who are your customers, and what is the biggest problem that you solve for them?

Shmuel Shottan: BlueArc’s customers are mainly focused on high performance applications and are in data intensive markets. To be more specific, for many of our customers, BlueArc helps in driving value creation and revenue generation–for example, drug discovery, computer generated effects, and design and simulation. These are all customers that appreciate the value created by deploying a BlueArc system, which accelerates their applications. Another example is focused on consolidation. By consolidating many storage islands through the deployment of a BlueArc system, the customers lower their total cost of ownership and simplify their infrastructure.

Sunshine: There’s been a lot of talk about “dedupe for primary.” What are your thoughts about this technology, and where do you see it going?

Shottan: Dedupe for primary lagged dedupe for backup. Dedupe for backup became an embedded feature in every VTL system. Dedupe for backups lends itself because you have duplication. Primary was the next wave, because everybody started to talk about OpEx–operational expenses–or how much power, cooling, and space it takes to store all the data, which was increasing by a factor of 10 for many enterprises.

Data has a lifecycle. If you look at it holistically, you can apply the 80/20 rule. Only 20% of data needs to be accessed at the kind of performance level BlueArc realizes, while 80% of data is rarely accessed after a certain time period. Yet, all of it must be available 24/7. We’re solving the problem by adding a virtual tier of storage that is online, but which is compressed, and therefore is 1/3 or even 1/4 the cost.

Dedupe for primary is more challenging than dedupe for backup. It lends itself less to repetition, because it’s all different files. In tape libraries, you can do 30x because it isn’t all backup. The easiest way for a storage appliance is file-based dedupe, but that is not very efficient. Depending on the data set, you might get 80%, or maybe as much as 50% efficiency. The technology most relevant is the one taken from the backup dedupe, variable block. So while the need was there, the ratios were not good enough to justify the compression in many cases.

Sunshine: What types of industry verticals are facing the biggest storage challenges?

Shottan: Some of the industries that need this kind of accessible data storage include: media and entertainment, oil and gas, and bioinformatics. The data that is being collected and the applications that manipulate it are huge. “Huge” is a technical term, of course. (Laughs.)

Our solution includes primary storage compression, for which we have partnered with Ocarina Networks. Ocarina represents the next generation solution, which goes beyond block-level or file-based deduplication, and is far, far more effective at increasing capacity.

Sunshine: What are the specific storage needs of these industries?

Shottan:  At a very high level, those are industries which have the following two requirements: Firstly they are data intensive industries, which means lots of primary storage needs. Secondly, their ability to efficiently run their business depends on how fast their applications run. This is why the two key attributes of the BlueArc system: performance and scalability, apply well.

Let me tie this need with the reason we have partnered with Ocarina. For industries such as oil and gas and bioinformatics, the situation is this: upon completion of a processing run, all the data needs to be kept around. However, for successive processing runs, the data can be compressed. This is where our multi-tiered storage comes in, and beyond that, we’ve been able to seamlessly integrate with Ocarina’s appliance to achieve this compression without having to invest in any new storage. Ocarina is the only offering we found that could successfully compress media rich files such as those created in genomics labs.

Sunshine: What about movies? Why are they so data intensive?

Shmuel Shottan: The biggest storage costs for the media and entertainment industry are in production. We work with studios that do animated films, and while the final product fits on a DVD, the production phase can be tens or even hundreds of terabytes. And it is dynamic data, not static data. The rendering time or the processing of rendering a scene to a movie takes time. And time is money.

For example, it can take almost a week to render hair on top of the head of an animated creature. Say you want to reuse that hair. If you as the animator has to recreate it from scratch because it’s now on tape, that’s weeks of extra work. This is why you want to keep that data online, because once you put something on backup–whether tape or VTL–you no longer can easily access previous scenes.

Sunshine: I’m guessing that for production, OpEx costs are extremely high.

Shottan: Yes, and we realized that this is a perfect situation for compression to Ocarina. They have designed compression algorithms for industry specific file types, including those used by Hollywood film studios. We’ve been able to integrate our two offerings and significantly reduce their storage footprint.

Sunshine: Thanks for taking the time to speak with me.

Shottan: It was my pleasure.

Shmuel Shottan is an industry veteran with thorough experience in the research and development of hardware and software, and in engineering management for firms ranging from start-ups to Fortune 500 companies. He holds a BSEE degree from the Technion, Israel Institute of Technology. His full bio is here.

BlueArc is a leading provider of primary storage solutions to enterprise markets, as well as such data intensive markets as electronic discovery, entertainment, federal government, higher education, Internet services, oil and gas and life sciences.

There is a definite feeling out there that storage blogging is growing. It’s a more connected, social world than ever before in the past, the result being higher quality content.

Here are some of the tidbits that jumped out at me this week:

Storage Monkeys, relaunched last month (just like this blog!) announced today that it plans to add more social features, with the “goal of being the premier social networking site for storage users, by storage users.”

Beth Pariseau at StorageSoup self-corrected in her discussion with and about Storagezilla’s post over the weekend regarding EMC’s scaled down version of Networked, Fast Start, which can soon be used as a virtual storage appliance. As Beth mentions in this same post, VSAs are a-popping from numerous vendors, and she links to a very interesting post by Storagebod in which he puts several of these products through their paces. Beth is one of that select group of journalists out there who really gets the value of bloggers and blogging.

Meanwhile, there’s a new group blog/magazine on the storage block–Gestalt IT, which brings together voices from storage and IT. Today’s post is by Stephen Foskett–a very active and positive force in the storage blogo-tweet-osphere community–which reacts to what he points out is a bit like the storage Oscars. That is, the announcement of the winners of the Storage Products of the Year awards from Storage Magazine. (Ocarina Networks was named a finalist this year.) Steve has a thoughtful response.

Here’s something that might surprise you–one of the sectors that has been hit hard by an upsurge in storage demands is movie studios. And while you’re probably imagining that video is the source of their storage woes, in fact this is not the case.

As it turns out, video is a relatively small part of the movie-making process from a data point of view. Rather, it’s the growing adoption of all-digital HD and film workflows, and the move to 3D (stereoscopic) production that has led to this new level of demand.

While the price of disk is dropping, storage costs for studios have nevertheless skyrocketed as a result of these new digitization technologies. In essence, every movie is now broken down into a series of digital stills, regardless of whether it is animation or live action. This has led to a situation in which the amount of data for just one 18-month shoot is unmanageable without some kind of compression solution in place.

This is a challenge that Ocarina has taken up, and I encourage you to take a look at how its next generation optimization solution has been deployed for this particular sector.

Learn more by visiting this page and downloading the “Digital Intermediate White Paper.”