The session takes place on Tuesday, March 8, 2011 at 8:00 am PT / 11:00 am ET / 16:00 GMT.

Register now

For fast and reliable long-term retention, integrate EMC Data Domain Archiver into your backup and archive environment today.

EMC Data Domain Archiver is the industry's first system for long-term retention of backup and archive. This single deduplication storage system can address your operational backup needs and long-term data retention requirements, eliminating the costs and complexities associated with tape-based backup and archive.

• Cost-effective scalability for long-term retention with up to 28.5 PB of logical capacity
• Fast, inline deduplication with up to 9.8 TB/hour throughput
• Minimized tape costs and simplified data management
• Easy integration into existing environments regardless of backup or archive software

• Enhanced replication management
• Enhanced reporting capabilities
• Role based administration/access control
• Improved remote management
• An improved configuration wizard
• All new functionality in DDOS is available via the CLI
• Enhanced SNMP functionality DD Boost with encryption for encrypted optimized replication with BE, NBU, and NW
• DD Boost load balancing and link failover enhancements
• Encrypted replication
• LACP support for increased performance
• Myriad and sundry smaller hardware, connectivity, and OS enhancements

All in all, this is a very significant release, and a very significant step forward not just for performance and capacity, but for the DDOS itself. 

Of course it didn’t take me too long to figure out that backup was far more complex than I initially understood.

Complex because it touched every aspect of our IT environment. From the client, with its associated OS and application, through the network with its myriad intricacies, to a server, running a database, a scheduler, policy engines, and massive storage. Agents were often required to get database backups that were useful (anybody remember paying hundreds of thousands of dollars for SQL Backtrack licenses because you really didn’t have any other good options for Oracle backup?).

Complex because the architectures and the software was complex. Complex because backup was operationally intensive. Complex because of the relentless pace of change and data growth in our IT infrastructures.

So while I feel nostalgia, I think it is the nostalgia of innocence lost. Nostalgia for the way things I naively thought should be. Not the way they really were. Really are.

In the last 20 years, backup complexity has nearly overwhelmed us. The number of applications that require live or hot backup has multiplied significantly both in number and scope. Sharepoint and Exchange. DB2 and Oracle. SAP and SQL. The architectural and administrative options most backup applications offer has grown dramatically. There are more protocol choices than ever before: CIFS, NFS, FC, OST, NDMP, and BOOST (and even, still, SCSI). Modern backup applications support a huge variety of operating systems, application clients, specialized application agents, and array integration tools.

Time (and perhaps a grain of hard won wisdom) has shown that complexity is growing. And I don’t have any reasonable expectation that complexity is going to diminish. All I have is my nostalgia.

With a few notable exceptions that is.

Data Domain systems are one such exception: they are extraordinarily simple and elegant both in design, and in terms of operational utility. (And, as an aside, I believe that this is one of the big reasons for their extraordinary success and acceptance with backup administrators and operators.)

So in the absence of a return to innocence or some revolution in backup, we are only left with a way to manage with the complexity. To cope with its operational impact. And to minimize the effect it has on our organization.

And to do this, it is almost mandatory to have a tool like Data Protection Advisor (DPA).

Because DPA may not be able to take the complexity out of backup, but it can take some of the complexity out of understanding and managing a complex backup environment. It is not a time machine that will take you back to the relative simplicty of backup as it was 20 years ago. But it can help drastically reduce the operational complexity of your current backup environment.

Now I recently visited a client who had some very direct, first hand experience of this. We had just finished a proof of concept of DPA in their VMware environment. And it was a big VMware environment: some thousands of VMs for test and dev alone (production systems are slated to find their way into a virtual environment in the coming year). Big enough that the client didn’t know what they didn’t know. And big enough they had no idea how big they had got, and how much or little of it they were backing up. They assumed they were doing a pretty good job. But they didn’t know for sure.

So we put in DPA. And we were pretty shocked. They had told us to expect about 1200 VMs. And they thought they were backing up about 800 of them, with a very high degree of success. In fact, we found about 2500. And the success rate was less than 25%.

(Another aside: DPA does not count situations in which there are multiple failures to get a backup in one window, followed by a success, as anything other than a success. So when you have a 25% success rate on a 1000 systems, that means you only got a backup from 250 systems. Not that you got all 1000 after a few retries.)

Without DPA, they had no ability to measure this, or report or analyze it. The backup application only told them what the backup application knew. But DPA ties into vSphere and reports can draw data about the private cloud. It knows how many systems you have, and how many are protected. And how many are not.

If you are running Avamar in your VMware cloud, it can report per Avamar domain (frequently different domains are used in for different tenants in multi-tenant environments). The reporting that Avamar has for intra-domain analysis can now be extended across domains, and across Avamar grids. Auditing and change control are also a lot easier to manage: the standard Avamar change report is extended to include the user that made the change. So when you have 1300 VMs that are not protection, at least you have an audit trail to determine who excluded them from the backup process. Of course, the next question should be why they are excluded!

A different client locally is struggling with a different problem: they have half a dozen Data Domain systems. They have deployed them for a variety of uses within their backup environment—Exchange, application servers, and database servers all have their data backed up to the pool of Data Domain storage. But they are out of capacity. So it is time to add more. Who is going to pay for it though? The Exchange team? Or the database team? With deduplicated storage, this is not a trivial problem to understand. How much of your deduplicated capacity is being used for a give application set? Do you know? The client has elected to use DPA to measure consumption of storage. So when it is time to acquire more, the biggest consumers of capacity can also make the biggest contribution to the acquisition.

As it turns out, this is actually a pretty critical piece of information to understand. In larger organizations, which leverage a shared infrastructure like backup, different groups consume different amounts of that shared resource. Who pays for what? What is fair? How do you ensure that politics takes a back seat to facts when it comes time to pay for additional resources? DPA offers a solution. How much is consumed, and by whom, is very easily understand. Either for Data Domain or Avamar systems.

Is disk contention causing bottlenecks in backup? Do you know? Your backup application can’t understand this, but DPA can.

Are link drops causing missed backups?

Has the proliferation of VMs meant that entire chunks of the infrastructure are not protected?

Has somebody just added a mass of data that has cause a spike in your deduplicated disk consumption?

Complexity.

DPA permits me to be just a little nostalgic, without begin naïve or putting my head in the sand. And it means that we can solve a few of our backup problems quickly, efficiently, and elegantly. And while it may mean, sometimes, that we have more problems than we thought we had, fixing a problem that we understand is always easier than fixing one we don’t.

So while I don’t have a time machine to take me back to a time when backups were simple, DPA can help wind the clock back just a little, and take some of the complexity out of the job.

The first is that you can take a data centre floor full of tape and stick all that data on a couple racks worth of disk. Take 12,000 tapes and put all that data on less than 400 disk drives. Literally conslidating some 800 square feet of infrastructure into 10 or 20 square feet. That is an easy visual I think--and what that speaks to the level of consolidation that is possible. Much like server consolidation, this approach can offer some truly dramatic savings in space consumption.

Secondly, I talked about the ease of replicating backups to a second site. Without deduplication this is so expensive (in bandwidth costs primarily) that almost nobody but a privileged few actually did this. I don't think it would be an exaggeration to estimate that less than 1% of customers fell into this group. With Data Domain and disk-based deduplication, we see the exact opposite: a large percentage of our customers replicate the systems to an alternate location. A completely different cost basis has enabled completely different behavior.

I went on to comment that as a result, for short and medium term backups, not only had data deduplication products become the technology of choice over tape for the majority of customers, it was doing so at a price point that was roughly equivalent to tape.

And at that point one gentleman raised his hand and said something like: "hey I am one of those C-level folks you were just talking about and I don't know much about backup other than as a line item in my budget--so tell me: is disk deduplication really cost equivalent to tape or is that just a line?"

I replied as honestly as I knew how and told him that it was absolutely not just a line. And I gave him some perspective on this.

Over the last four or five years I think a reasonable consensus has emerged regarding the cost of disk based backup.

Initially, when all we had were VTLs without deduplication it was very difficult to justify disk backup with more than one or two weeks of retention. It was too expensive and tape was just so much less expensive for longer retentions (as the cost of the disk capacity and infrastructure rapidly added up with increased retention periods). Nevertheless, virtual tape libraries were very popular as they solved many of the problems associated with tape: reliability, performance, speed of recovery, manageability, and so on. As a result, not too many people had a problem cost justifying VTLs, as long as only one or two weeks of data were going to be retained.

And that was the state of affairs for a couple of years.

Then we added deduplication to the mix, and there was another change. The second wave of widespread adoption of disk as a backup target began. Disk based targets became cost justifiable for retention periods of, on average, three to six months. Perhaps as much as a year in some cases. Beyond that, the cost advantages of tape again become too significant to ignore. And as a result, in most cases, this longer term retention remains the domain of tape. But for shorter periods, disk with deduplication gives us all the benefits of disk: including reliability, inexpensive replication.

Fair enough.

The final part of my answer dealt with the one major assumption I make when I say that disk is cost competitive for short and medium term retention: that when we do the TCO analysis you are considering an upgrade or replacement of your tape based infrastructure in the next 2 or 3 years. Because if you are not, and your current infrastructure is meeting your needs then it becomes very difficult to cost justify buying something else. (Notice that I said cost justify. There are still a lot of other good reasons to move to disk than just cost: better reliability, faster backups, better DR, lower operational effort, and so on).

But assuming that at some point you are going to outgrow your current tape environment then I think it is pretty easy to justify the move to disk with deduplication.

So with that said I think it is very reasonable to conclude that you can deploy disk with deduplication for backup at a total cost of ownership that is equivalent to tape or less.

Which leaves tape only the use case of the remaining very long term retentions which are not cost justifiable on disk. For now, anyway.

But there is another important reason to move to all disk: cost. I will have more to say about the TCO of disk backup solutions shortly, but one of the most striking, and perhaps counterintuitive things about D2D2D is that it usually has a better Return On Investment (ROI) and a relatively lower TCO than D2D2T. That is to say: using disk for off-site rather than tape is usually easier to cost justify than using disk for onsite.

Now there are a thousand reasons for the success, not least the astounding level of execution in the integration effort and the comparative ease with which the Data Domain and EMC teams joined together. But another key part of the success has been the technology. Simply put, the inline, CPU bound methodology is beginning to demonstrate the real strength of this approach.

I think it is fair to say that, at least from the perspective of an outsider, it wasn't always obvious that the Data Domain strategy of putting the CPU inline and deduplicating everything before it lands on disk was the best approach from a performance perspective. After all, this approach initially offered only modest performance. But over time the performance has improved. A lot.

In the last seven years the performance of the Data Domain appliance has improved by nearly seventy fold: from a relatively modest 30 MB/s to a rather extraordinary level of over 2000 MB/s.

I like to think of this performance improvement in terms of the two curves that have determined the course of virtually every storage technology for the last twenty years or more. They are the Intel curve and the Seagate curve. The Intel curve is described by Moore's law: performance of CPUs will double every 12 to 18 months. The Seagate curve says that the capacity of a disk drive will double every 12 to 18 months. Unfortunately it has a rather less desirable corollary: performance as measured in IOs per second and throughput will double not every 12 to 18 months but every 5 to 10 years.

Which is why we have monstrous amounts of data but backing it up and recovering it, with either tape or disk without deduplication, is just as hard today as it was 10 years ago. Harder perhaps. Data capacities have increased by 20 to 50 times in that period, but throughputs have not. And as a result if you have stuck with a traditional back technology like tape or VTL you have to keep devoting ever larger amounts of infrastructure with ever larger amounts of capacity to backing up your data.

Your data is on an Intel curve.

Backup infrastructure is on a Seagate curve.

Or rather, it was. Until Data Domain came along. And achieved something quite extraordinary.

And that is increasing both performance and capacity on the Intel curve. Performance increases because fundamentally, the most important (and virtually the only significant) bound on performance in a Data Domain system is the CPU itself. And capacity increases by virtue of the combined impact of the Seagate curve and the multiplier of deduplication. The consequence: both performance and capacity have increased by 70 times in the last 7 years. Which is simply unprecedented.

Now, if you are anything like the vast majority of customers I talk to, if you are anything like the customers that EMC and IDC surveyed recently your data is probably growing at about 62 percent annually.

So now we can see why having an inline CPU-centric approach is so important: because it is the only way to keep up with data growth.

If you choose a backup and recovery technology that is post process, disk centric rather than CPU centric, you are choosing an infrastructure that inherently cannot grow with your data. An infrastructure that can only grow performance by increasing the number of spindles. And that is an infrastructure that will become increasingly expensive to own and operate as the gap between your data (on an Intel curve, remember) and the inherent capabilities of the technology (on a Seagate curve) increases.

Over time the only way to narrow this gap will be to add additional units to the infrastructure. That which requires 2 systems for this year will require 4 systems in 18 months and 8 systems in 3 years.

From a cost of ownership perspective this is exactly the wrong direction to be proceeding in.

The CPU-centric approach offered by Data Domain systems is unique in that it is the only approach that delivers infrastructure that grows in performance and capacity on an Intel curve. And it is therefore unique in the inherent capability of the technology to deliver infrastructure that can grow as fast as data grows. Which should mean that the approximate cost of Data Domain disk infrastructure dedicated to backup and recovery will not grow at a rate greater than the organic growth in your storage infrastructure. It won't require 8 times as much infrastructure just to keep pace with organic data growth, as other architectures might.

As a final note: speaking to the total cost of ownership of a backup environment is difficult: it includes hard and soft costs. Costs for infrastructure, power, cooling, labour, software, maintenance, and so forth. Properly speaking therefore, my conclusion speaks to just the infrastructure component, and Data Domain's ability as an infrastructure component to grow in capacity and performance at a rate approximately equal to the average rate of growth for data. And while infrastructure is an important component, it is not the only component. Therefore, to generalize my more limited conclusion regarding infrastructure to one including the entire total cost of ownership is not, strictly speaking, a valid generalization.

Increasingly, the new field of battle is in deeper integration between backup software and dedupe storage. Avamar started this wave a long time ago, and it remains the most advanced implementation. More traditional backup software is also moving in this direction:

• Symantec pioneered the idea using OST and Data Domain Boost to integrate NetBackup, and now Backup Exec 2010 with Data Domain and other OST licensees.
• Starting this summer, NetApp and SyncSort are offering channel-based integration packages for SyncSort to manage snapshots on FAS systems. Not quite Boost, but reacting to the same problem.
• Also starting this summer, Symantec are offering their own storage systems in some geographies for PureDisk. (I love OST, but how weird is it to be on the NBU 5000 team in the “No Hardware Agenda” company? It would be like working in DB2 support at Oracle.)

As noted below, EMC has moved aggressively to integrate NetWorker and Data Domain based on the Boost technology at the Storage Node (backup server) level. This is now available in the latest release of NetWorker. Avamar and NetWorker have already been integrated for some time from the NetWorker client.

Integration brings speed, as discussed below, but it also brings lower OpEx. Following up on the launch last May of NetWorker using Boost to integrate with Data Domain, we looked in detail at some of the results in our beta sites. To these customers Data Domain was already fast, but being able schedule replication, manage multiple retention policies, clone to tape as well as monitor and report on Data Domain systems all within NetWorker also brings significant efficiency improvement. One customer said they expect to see a 20-30% reduction in management time.

See prior memo.

DD Boost is distributed software. Part of it runs on the backup server of traditional backup software. The other part of it runs on the Data Domain system. It off-boards the math of the Data Domain dedupe process (cutting the data stream into variable-length segments, fingerprinting them, and if they are new, compressing them) to the backup server. Lookups of what’s new versus old is done in batches on the backing Data Domain system.

• This makes the backup server 20% less loaded (less data to copy).
• It makes the LAN between the backup server and the Data Domain system 95% less loaded (sending only deduped compressed segments).
• It gives the Data Domain system 50% higher aggregate backup throughput compared to our prior Symantec OpenStorage benchmarks, or about 2.4x faster than the VTL or NFS benchmarks on the same system.
• It includes management interfaces for backup software to control the Data Domain system, including managing per-file replication.

The performance of all Data Domain systems is being restated this week based on DD Boost, our new fastest protocol. We didn’t say it at the time, but the recently announced Data Domain Global Deduplication Array (GDA) software is based on extensions to the DD Boost technologies. It will support DD Boost implementations across the industry. The GDA performance doesn’t need restating. But the DD880, for example, just went from a prior benchmark for aggregate backup speed of 5.4 TB/hour to 8.8 TB/hour.

NetWorker will integrate the distributed DD Boost library in the second half of 2010. It will then have caught up with NetBackup in Data Domain integration and speed. It will have all the same control over Data Domain replication that Symantec products have had (or, in NetWorker terminology, Cloning Controlled Replication.) This caps a long list of fundamental improvements NetWorker has focused on for the last couple years to optimize for disk-based backup, especially to dedupe storage. NetWorker now offers best of breed technology for dedupe at both the NetWorker Client and through the Storage Node server.

• NetWorker has integrated the Avamar client code in its clients for some time. By policy, if an admin wants dedupe to happen at the client, the Avamar process may be invoked, interacting with an Avamar storage system in the datacenter.
• With these announcements, NetWorker now integrates through DD Boost in its Storage Node server. Data Domain will be seen as a standard storage device type.

What does this all mean?

In the future, most backups to Data Domain systems will be through DD Boost. Using the OpenStorage distributed integration method, Symantec and Data Domain proved a while ago that the combination of backup software and dedupe storage could be made more manageable. Now, it’s also faster and more efficient. By a lot. The only reasons not to use DD Boost moving forward are that you need to use fibre channel (FC) or that the backup software doesn’t have a DD Boost integration program. Since LAN bandwidth is made so small with DD Boost, a lot of admins can consider LANs where before FC was the only technical alternative. Backup vendors have been asleep at the switch on competing with OpenStorage. They won’t make that mistake for too much longer.

Remember when Symantec first partnered with NetApp after EMC bought Legato? Their pair-wise effort created some of the Ur-structures that underlie OpenStorage, but it took a while to generalize the interface. We are in the same maturation process between Data Domain and NetWorker teams. Those lessons will result in a clean set of interfaces, but it will take a little time to complete.

But don’t misunderstand: our engineering relationship around Symantec is still critical to us. This is one of those times when it serves our mutual customers hugely to have us collaborate, even if some of our products can also compete. Our respective teams have continued to have strong links, and we wouldn’t have it any other way. DD Boost is the new name of the Data Domain OpenStorage software option because (1) DD Boost supports more than Symantec OpenStorage now, and (2) it does a lot more than it used to. But the interaction of DD Boost with NetBackup and now Backup Exec is through the Symantec OpenStorage APIs, and new features will continue to be developed in tandem as long as they allow us to participate.

In the short term the advantages to Data Domain customers using DD Boost-linked software (EMC or Symantec) are significant and differentiating. At some point, the other backup software vendors will recognize how important it is to catch up on these features. You can estimate the relative R&D focus of these vendors by timing how long it takes for them to support OpenStorage-like interfaces. We’re happy to help them figure it out. In the meantime, DD Boost is already supported by more than half the enterprise backup market.

You can’t do a DD Boost style approach with a post-process dedupe system. Data is deduped live, on the way in, inline. If they ever published their internal dedupe throughputs, post-process vendors might be asked to compare to Data Domain’s. That was already awkward; now they might never do it.

Even inline vendors such as IBM are going to have to gulp a little. EMC now offers a complete dedupe storage system (the DD880) that is more than 4x the backup speed of their comparable single-controller ProtecTier gateway (rated at 500 MB/sec., or 1.8 TB/hour). Their real systems, appliances with disk attached so you can review price-performance apples/apples, are considerably smaller and slower.

Unlike pure software vendors trying to distribute dedupe, the EMC approach includes complete dedupe storage systems, so performance and management are predictable, simple and scalable. We’re focused on developing end-to-end solutions. We can imagine things differently because we develop both the software and the storage. Remember how in the 90s some used to think LUN management would be done on servers? It didn’t scale, and in the end, serious implementations use arrays to do this. Systems have already won with dedupe as well; some vendors just didn’t seem to get the memo.