Enterprise Storage Strategies

What Do We Call This Cloud Storage Thing?

sfoskett — Wed, 31 Mar 2010 20:44:00 GMT

Not all cloud storage is equal, but it's even difficult to talk about those that are more comparable. Does an API make it a platform? Are all APIs the same? Setting these questions aside temporarily, we come to a more fundamental question: What should we call cloud storage services, anyway?

The Case for "Platform as a Service" (PaaS)

Platform as a Service is variously defined, but it generally refers to a middle layer between infrastructure (hardware and software) and applications. Cloud platforms offer simplified interfaces for a new generation of applications that use web protocols and are often coded in languages like php, Java, .NET, Ruby, Python, and the like. Public cloud storage solutions like Amazon S3, Nirvanix SDN, and AT&T/EMC are fundamentally different from traditional storage infrastructure.

Both control and access use a web services (REST) API rather than a traditional block or file storage protocol.
These APIs are really programming interfaces, and libraries make them highly accessible to modern web-oriented languages.
To varying extents, these systems do a lot more than just storing data - they can organize it, manage it autonomously, and even process it!*
Cloud storage APIs often allow rudimentary programming of data control and disposition.
Public services like these are multi-tenant and have much more extensive accounting and reporting than any conventional storage system.

In short, public cloud storage services match the letter and spirit of Platform as a Service definitions like the one at Wikipedia. * Did you know that Nirvanix can even transcode media in the cloud? Cool!

The Case for "Infrastructure as a Service" (IaaS)

No storage system is a programming environment, and even the best public cloud storage service isn't anything like what is generally understood as PaaS. Find 10 people who think they know what PaaS is, and few will name Amazon S3 (the most visible public cloud storage service) in their examples. In fact, most won't even think of storage at all! Storage is infrastructure and always will be. The new REST access methods are great, but does an interface change the nature of a system? There are REST APIs to control old-school non-platform storage, too, and vendors might tack on API access at some point. That Wikipedia article about PaaS goes on and on about programming and application development, making scant mention of storage. It's not a platform, so it must be infrastructure!

The Case for "Something Else as a Service"

Storage really isn't PaaS or IaaS. Those definitions are all about compute, and storage is different enough to warrant its own definition. Whether it uses a web services API for control and access or uses old-school block or file mechanisms, storage as a service is a new business model for an old commodity. So what do we call it?

NetApp and Iron Mountain like "Storage as a Service", abbreviated as "STaaS". But that's also used for "Software Testing as a Service".
SNIA suggests "Data Storage as a Service", abbreviated as "DaaS". But that sounds awfully like "DAS" or "direct-attached storage", exactly the wrong connotation!
How about we ditch the "XaaS" format entirely, as Sam Johnston suggested, and just call it something like "managed storage services". But that's exactly what the old managed storage providers called their offering a decade ago!

What do you suggest?

Call it Anything

In the end, it really doesn't matter what it is called. Cloud storage is here in many guises: Hardware or software products, purchasing models, and novel services alike. Soon, it is likely that the "cloud" moniker will lose its luster, too, but that won't change the core value proposition that some storage solutions bring to the table. Certainly this entire discussion will be forgotten as the market adopts some name or other for the thing. And that's for the best.

Could Cloud Exchanges Work For Storage?

sfoskett — Wed, 03 Feb 2010 15:27:00 GMT

CNET's Gordon Haff wrote a great piece on the shortcomings of the exchange model for cloud computing. His prognosis is right there in his title: "Why cloud exchanges won't work." I've done some thinking and writing on the topic, and it's easy to see Haff's point: Interoperability, security, and inertia threaten to derail this new concept before it starts.

Shortcomings of the Exchange Model

Haff's concept is centered on the following three simple qualifiers for cloud exchanges:

Any platform involved in an exchange must be compatible, allowing a workload to seamlessly move between interoperable systems. This is both critical and absent with many of the cloud computing services available today. Most are incompatible on a basic level, using different hypervisors for example. No cloud exchange can seamlessly move an EC2 Xen instance to a Terremark VMware environment, although Rightscale is working on clever translation systems.
Not all service providers are equal when it comes to security and compliance, either. I've often bemoaned the fact that so many service providers are not enterprise-ready, and this will be even more of an issue with an intermediary deciding where to run a particular job. How can a buyer be sure his workload will be safe?
Haff also points out that a compelling service must be cost-effective, and certain elements stand in the way of this. He questions the value economies of scale will bring to very-large service providers. He also wisely points out that additional costs to move supporting stored data could derail the return on investment

These are not intractable problems, but they are real concerns. The issue of portability is especially thorny for cloud computing, as vendors focus more on basic functionality and innovative features than compatibility. Yet one can envision a future in which even these issues are resolved: Haff worries about service providers moving "up the stack", but this is exactly where compatibility is likely to emerge. It is easier for me to imagine a number of interoperable Java or .NET platforms than truly compatible Xen environments!

It also seems that the current efforts to define cloud service description and provisioning APIs addresses many of these concerns. If a standard API could specify compatibility, the network environment, and security requirements, an exchange could offer a wide variety of service providers with these capabilities. I imagine an airline model, where not every airport is served by every airline, but there is enough competition even at the fringes to keep the resellers viable.

I am also much less concerned about return on investment than Haff. I have seen amazing economies of scale achieved at Nirvanix (where I am Director of Consulting) and can imagine these in many areas. Cloud service providers are driving cost out of many areas by standardizing and centralizing management operation and provisioning as well as hardware and environmental costs. As cloud providers set up shop in super-efficient data centers and train highly focused management staff, they are likely to surpass the economies of even the largest abd best-run end user environment.

What About Storage?

As I wrote in December, spot pricing for cloud storage is much less attractive due to the sheer inertia of data. But an exchange model might actually be attractive even after this is taken into account. The major public cloud storage providers have already moved up from the infrastructure (bytes and blocks) to the platform (object) level, though the "ammo provider" private cloud market remains focused on the former. And many efforts are already underway to create basic standard interfaces for both provisioning and access across these major vendors.

Let's take Haff's concerns in order:

Interoperability of cloud storage is likely to come well before compute thanks to the more constrained workload involved. While a compute platform could be asked to perform almost any task, storage in general is focused on a simple usage model sometimes called CRUD: Data is created, read, updated, and deleted. This has already led to a proliferation of pre-standard or de-facto standard generic interfaces to multiple cloud storage services. SNIA's rapid work on a standard cloud data management interface shows that interoperable public cloud storage isn't that far off.
Security and compliance is much less standardized among private and public cloud storage providers. I believe that all enterprise-focused public cloud storage providers should focus their efforts on offering solid, reliable, and secure systems, but this has clearly not been the case universally. And although efforts like the SNIA CDMI would standardize provisioning of services, we will need a much more robust vocabulary to specify the level of security and compliance required for a specific application. But not every application is right for cloud storage anyway. Surely a minimum standard for security can be agreed upon by multiple providers, allowing at least some bulky applications to use cloud storage without worry. We will eventually develop a more complete mechanism which will allow more sensitive applications to use a cloud exchange.
Cost effectiveness is another thorny issue. Public cloud storage for the enterprise isn't a race to the bottom in terms of cost; it has to be about more than just cheap capacity. Indeed, the public cloud storage market is already splitting into three categories: Cheap personal storage and backup, inexpensive storage for developers and web applications, and feature-rich enterprise-grade offerings for businesses. But cost will always be a factor, and cloud storage must prove its value. An exchange that resulted in higher prices or surprise fees to move data wouldn't be a success. This last is worth noting: Since moving storage between providers will always require time and costly bandwidth, I expect such an exchange to focus on net-new data, not migration of existing capacity to a cheaper provider.

Cloud storage is significantly different from cloud compute, and spot pricing and exchanges might make more sense for data. Indeed, many of the concerns voiced by CNET's Haff are less troubling in the storage world. But the issue of return on investment remains: Could a business cost-effectively use varying cloud storage providers? I suspect some will soon set up shop and try to find out!

Cloud Integration Mixes CommVault Simpana Management With Cloud Storage

sfoskett — Mon, 01 Feb 2010 14:33:00 GMT

Since cloud storage is not backup in and of itself, anyone wanting to use cloud for backup needs additional software smarts. One of the smartest data protection and archiving suites out there is CommVault's Simpana. Mixing their intelligent software with the benefits of cloud storage for data protection makes a killer combination!

This is precisely what CommVault announced this morning. CommVault's cloud integration brings seamless access in Simpana 8 to public cloud storage from the leading providers as well as local tape and disk. CommVault's traditional strengths in archiving and data management are magnified by the addition of cloud storage. For example, customers can use Simpana to identify and move certain data types to off-site cloud storage for enhanced protection and availability and to take advantage of the lower cost and higher flexibility and scalability of these services. Or they can use the cloud as a disaster recovery location, saving weekly copies to the cloud and leaving everything else local.

CommVault's Simpana includes powerful encryption, data deduplication, and enhanced indexing capabilities, and these work great with cloud storage. The first two will ease two concerns many have voiced about public cloud storage: Privacy and performance. Encryption kills deduplication and vice versa, but by integrating these functions the negatives are removed. These save money, too, since less data is stored and no additional hardware or software is needed.

But it is CommVault's indexing capabilities that really get my imagination going. Simpana knows exactly where all data is stored, whether it's on local disk, offline tape, or online cloud storage. This alone will improve efficiency, but let's take things further. Cloud storage is smart, and capabilities can eventually be moved into the cloud. Imagine a protected deduplicated archive that is accessible on demand. This would be awesome for business intelligence, electronic discovery, or disaster recovery. Future cloud services could really make hay with this capability!

This support is thoroughly engineered and integrated, and CommVault will soon support all of the major cloud platforms. They announced Amazon S3, Microsoft Azure, and Nirvanix, and their logo page shows Iron Mountain and EMC Atmos, too. This broad support should help cloud storage adoption. Simpana can easily copy data wherever customers feel most comfortable hosting it, whether it's on-site tape or disk or a public cloud service or any combination of the three. There is no need to install any other software or roll out any hardware - it's all built into Simpana.

This kind of integrated solution is a major step on the way to the storage revolution I've long demanded. An archiving system that can intelligently manage data and move it between disk, tape, and cloud is one thing, but this one really makes use of the programmable and portable nature of cloud storage. I'm looking forward to the creative solutions customers come up with in the near future, as well as the advancements introduced by companies like CommVault!

Read more over at Network World, DGIC, or my personal blog!

Mr. Backup Is Right: (Cloud) Replication Is Not Backup, But Backup Is!

sfoskett — Fri, 22 Jan 2010 16:01:00 GMT

Go read that headline again: W. Curtis "Mr. Backup" Preston points out on his blog that replication is not backup, and we can't disagree. Keeping alternative copies of data in multiple locations is a great idea, reducing the risk of data loss and potentially enabling enhanced access, but it's not a historical data protection (aka, backup) strategy. Backup requires management of multiple historic copies of a data set. Clearly, cloud storage in itself isn't backup.

Backup vs. Storage

SNIA defines "backup" thus:
[Data Recovery] A collection of data stored on (usually removable) non-volatile storage media for purposes of recovery in case the original copy of data is lost or becomes inaccessible; also called a backup copy.
To be useful for recovery, a backup must be made by copying the source data image when it is in a consistent state.
[Data Recovery] The act of creating a backup. See archive.

Backup has always been a challenge for corporate IT. It's not "in the critical path", affecting the daily activities of business users and customers, so it usually gets short-shrift when it comes to financial and organizational support. Yet the ability to restore data quickly becomes job one for IT when it is lost or corrupted. I think Preston spells it out wonderfully in the first chapter of his (updated) seminal book, (UNIX) Backup and Recovery. Systems always fail, data is always lost, and having a good backup is the surest way to recover.

Storage industry folks have been suggesting that new technologies eliminate "traditional backups" ever since there has been an industry to speak of. Some of these technologies (RAID, replication, high availability, hash-based integrity checks) are great innovations in keeping online data alive, but they fall flat when it comes to data corruption. Others (mirroring, snapshots, versioning, CAS, CDP) are great at retaining multiple copies of data, but even these aren't true backup solutions. Good backup is much more than mere data protection: Backup must manage data, not just protect it. No basic storage technology will eliminate a real backup solution.

Skim through Preston's book (the index is online at Amazon!) and you'll see that merely creating and holding a copy of a given data set is just a small part of a real backup solution. These copies must be tracked, managed, and expired. Operating systems and applications must be integrated into the solution. Bare-metal recovery, disasters, and compliance must be considered. Storage folks ignore these hard-learned lessons at their peril, and any storage vendor who says backup is dead is revealing their ignorance or naïveté!

Cloud Storage For Data Protection

Although storage technology will never be a full answer to the data protection quandary, it has a lot to offer when it comes to assisting backup solutions. Disk technology has literally transformed the backup world in the last decade in the form of replication, snapshots, CDP, virtual tape libraries, and deduplication. These technologies give powerful new capabilities to the existing backup frameworks, overcoming the dismally-limited tape cartridge approach of the olden days. A state-of-the-art backup solution now relies much more on disk-based storage systems than tape or optical capacity, and many use disks exclusively.

Cloud storage presents new opportunities to enable more effective and efficient backup solutions. Most cloud storage platforms can be very highly utilized, reducing system cost, and can be flexibly and non-disruptively expanded as capacity needs grow. But some cloud storage systems go way beyond this:

One of the hallmarks of public cloud solutions is their physical distance from the systems that use them, decreasing the likelihood of data loss from a local disaster. Backing up to a site hundreds or thousands of miles away has long been a dream of IT, and cloud storage makes this possible and even cost-effective!
A few cloud storage platforms offer integrated policy-based replication of data (ahem, Nirvanix), and this additional geographic distribution further reduces the risk of data loss in a disaster. It can also aid in recovery, since data can be available locally at remote locations!
Like all disk-based backup targets, cloud storage is online and accessible, making restore operations quicker and easier. There is no need to wait for tapes to be recalled, delivered, located, and loaded when data is on random-access disk! But unlike local disk, public cloud storage can be accessible remotely as well, bringing this ease to distributed businesses and disaster recovery operations.
Cloud storage systems can embed metadata with stored content, further accelerating restore operations for systems that can use it since indexes no longer have to be rebuilt. This also enables new archiving and content management features, elevating backup to serve a primary business need.
One of the hallmarks of cloud storage platforms is their API-based programmability. Backup and archive management companies are discovering the ease and power of integrating programmable cloud storage right into their applications: Watch this space for announcements!
Further storage smarts are being embedded into cloud systems, too. We have seen deduplication and compression (check out Nirvanix partner, Ocarina!), data protection (Partners, Tarmin and Atempo), media transcoding, indexing, content distribution, and more.

Backup Is Backup

Simply adding basic data protection techniques like snapshots or replication to a storage system doesn't make it a backup solution. Storage isn't backup, but backup is! Backup systems can leverage storage capabilities, but a backup management solution will always be required to get complete data protection.

Clearly, cloud storage isn't a backup solution. But as this unique combination of capabilities demonstrate, it's much more than simple storage capacity. Like so many storage technologies before it, cloud storage is an enabler for advanced backup solutions.

SSPs, cloud storage providers, and internal clouds: Zebras, Giraffes, and Horses

sfoskett — Mon, 11 Jan 2010 14:25:00 GMT

My youngest daughter used to have trouble with her animals. Whenever she saw a giraffe, she would say "zebra" and whenever she saw a zebra she would say "giraffe!" Although an adult would never make that mistake, one can understand why a child would: She was new to these names, and they were entirely arbitrary words. Besides, both are quadrupeds with bizarre coloration and patterns. But my daughter definitely knew a horse when she saw one!

Private Clouds and SSPs: Horses and Zebras

Today's cloud storage world can be equally confusing to the uninitiated. Long-time IT folks remember the storage service providers (SSPs) of a decade ago and have watched as storage and server virtualization have gained prominence. When cloud storage began to get some press a couple of years ago, it was natural to try to fit it into the existing paradigms and understandable to fail to spot the differences.

Internal IT systems have been on the road to virtualization for years. I recall being excited about the potential of server and storage virtualization over a decade ago. But storage service providers like StorageNetworks (where I worked in 2000) didn't use any of this fancy stuff. Rather, the SSPs of last decade were zebras, built of the same storage area network (SAN) storage systems used by their customers but clothed as a service-oriented business.

As Hu Yoshida of HDS points out in his blog, SSPs were not a raging success. But they were not the colossal failure many assume - just ask the vast assortment of StorageNetworks alumni now in charge at places like HDS! True, all that conventional enterprise storage gear had trouble with multi-tenancy, but the real issue for SSPs was financial. They built out world-class storage networks (pardon the pun) and wrapped them in expensive home-built management and provisioning software. By the time they had a workable offering, the price tag had risen to levels that were hard to justify.

Today's private storage clouds, as touted by HDS, NetApp, and others, are horses of a different color. Enterprise storage systems are much more flexible and sharable thanks to integrated virtualization and advanced management features. But they retain the traditional enterprise storage access mechanisms, construction, and cost. If you're looking for a horse, it's a blessing to find one: These "cloudy storage" solutions can be plugged in and used precisely because they are conventional. The kind of storage workload described in Hu's blog (massive I/O connected to virtual servers) is best served by these impressive but ordinary storage devices.

Public Cloud is a Different Animal

Not every workload needs a workhorse, however. Many have questioned how widespread the need is for the next generation of high performance connectivity. Database systems remain the only really common high-I/O workload, though highly-concentrated server virtualization systems will also soon join this club. But these make up only a small percentage of overall IT server and storage deployments. The majority of applications demand low cost and high flexibility more than extreme performance.

This is especially true of the types of applications using public cloud storage today. Regardless of industry vertical, every business would benefit from having their vast reference and archival datasets available online rather than moldering on tape. Intelligent cloud storage platforms are rapidly being integrated with the best data management and archiving applications to make this a reality. Already, companies like Nirvanix are hosting petabytes of archival data for the largest corporations and governmental entities. They chose public cloud storage over internal disk or tape because it was competitive on cost as well as being exceptionally available.

The next generation of business applications will make use of the other big benefit of cloud storage: Collaboration. Forward-thinking businesses have already deployed applications with integrated data sharing using public cloud storage to enhance their technical support and customer service activities. A new wave of similar business-to-business collaboration tools is on the way. IT infrastructure folks might not have noticed this shift in focus by developers, but the revolution in collaborative software is about to strike.

Although the casual observer might not discern it, public cloud storage solutions are as different from virtualized internal systems and the old SSP offerings as giraffes are from horses or zebras. Today's cloud storage providers are rejecting conventional enterprise storage devices in favor of software solutions based on commodity server hardware. Server and storage virtualization, Fibre Channel SANs, and even enterprise NAS are rare in the data centers of cloud storage providers. Instead, they have re-thought the challenge of protecting data and servicing customers and their solutions have the side effect of being much less expensive. Hardware cost is disappearing, and cloud providers are instead focusing on the "glass floor" of operations and management costs, as well as raising the bar on service and availability.

Place Your Bets

This new world will not erase the old, but cloud solutions have little use for traditional server and storage infrastructure approaches. There is a race "up the stack" as IT companies deploy platforms and services rather than merely offering faster versions of last year's server and storage equipment. This is the reason for VMware's acquisition of SpringSource (and perhaps Zimbra) and a series of investments on the part of EMC, Cisco, Dell, HP, IBM, and the rest. There will still be a market for the nuts and bolts products that support traditional IT systems, but these big players are betting that the action is elsewhere.

What Will 2010 Bring To Enterprise Storage?

sfoskett — Thu, 07 Jan 2010 19:33:00 GMT

I'm loathe to give predictions, preferring introspection and outright silliness. But the turn of the year is a time of optimism, so I will take my turn at the megaphone to dish out some ideas I believe will come to pass in the coming year.

2010 will be a year of normalization (“righting the ship”) for enterprise IT : We will see a return to investment and building out new features after a year of financial panic. IT will begin again to focus on what they do well and continue to outsource everything else – including non-core applications. Without the threat of financial doom, IT folks will be willing to take more risks than in 2009, feeling that their jobs are no longer on edge.

With regard to enterprise storage, I think a few trends are particularly interesting:

Increasing virtualization drives higher I/O demands – VMware vSphere and Microsoft Hyper-V can now push the big I/O required by databases and other taxing applications, and these will be virtualized (finally) in 2010. This in turn will demand more storage I/O, so we’ll see increasing use of SAN arrays even at the low end of the market.
Expansion of SANs for SMB – As smaller environments (and smaller apps within large environments) virtualize more, they’ll start looking for intelligent, higher-performance SAN storage. This means a bonanza for vendors of iSCSI and sub-$20k storage devices!
Increasing use of archiving – Businesses of all sizes are interested in archiving for compliance and data management reasons, so the use of archiving software, hardware, and services will explode. I expect managed archiving services to be particularly interesting since this has never been a core focus of IT.
Pushing up the stack – Every area of IT is “moving up the stack” with tighter application integration, and this will continue as new technologies come to market. I expect special-purpose storage solutions (software, hardware, and services) integrated with applications (like Exchange, SharePoint, SAP, etc) to be a real focus for 2010.
The end of FC disks – Flash and automated tiering will combine with SAS to spell doom for traditional high-performance disk drives. We’ll see array vendors switch en-masse to larger capacity drives with SAS and increasing amounts of cache RAM and flash in storage systems throughout 2010. 2011 and beyond might see the end of high-performance disks altogether as SSD becomes entrenched.
The advent of extreme tiering – We’ll see flash, SAS, and cloud storage combined into super tiered storage systems, with a number of solutions appearing to cache, balance performance and capacity, and replicate data off-site. Virtualization will meld with cloud front-ends and automated tiering to become extreme tiering devices. This won’t be mainstream until 2011 at the earliest, but it’ll start happening this year.
Still not the year of converged networks – Although Cisco, EMC, and the rest will push hard for 10 GbE, DCB, and FCoE, it will not make a significant impact in IT spend through 2010. But 10 GbE will be deployed successfully in high-I/O environments (see number 1). ISCSI will continue its quiet rise, though.

What do you think 2010 will bring?

Can Spot Pricing Work For Cloud Storage?

sfoskett — Mon, 14 Dec 2009 17:29:00 GMT

This morning, Amazon took a step with EC2 that many had long anticipated: They announced spot pricing for cloud compute instances. EC2 customers can now name their own price, and Amazon will bring compute instances up at variable discount prices according to these "bids". This complements their March 2009 move at the other end of the spectrum, extra-cost reserved instances. This evolution of the cloud compute market was predicted by many, but Amazon deserves credit for making it happen.

The obvious next question is whether spot pricing can work for cloud storage. Although there are cases where spot pricing might make sense, the nature of data storage trends against this sort of pricing model. Let's explore the question further.

Cloud Compute

Cloud compute is obviously different from cloud storage: The former deals with running applications and the latter with storage of data. That the nature of usage of the these services is so different often comes as a surprise. Compute instances tend to be peaky, coming online, spiking, dissipating, and shutting down quickly. One can imagine cloud compute instances matching the graphic below, taken from my own educational slide deck, where cloud compute instances peak daily and fall off each night over the period of a week.

The value of cloud compute is illustrated clearly here: Rather than buying 2/3 the CPU capacity, as in a virtual infrastructure, this workload can be satisfied with just 1/3 the capacity by leveraging a cloud compute platform. Your mileage may vary, of course, but this real-world example shows why so many are perfectly happy to buy compute capacity on demand.

This also illustrates why spot pricing is so attractive to the service provider. The load on their infrastructure undoubtedly matches this pattern, with peaks and valleys over short periods of time. Charging a premium for reserved instances attacks the peaks by scaling up capacity, and offering discount non-reserved spot instances fills the valleys. Although the Priceline-esque "name your own price" concept might turn some off, it makes good sense in cloud compute.

Note that Amazon's spot instance pricing scheme is not quite this straightforward: They compute a spot price based on bids for compute instances in each region. Any bids below this threshold will not be run regardless of available capacity, so it is wise to monitor the current prices and not under-bid!

Cloud Storage

Cloud storage is different. Most cloud storage use cases involve long-term retention of large amounts of data. Imagine a gallery of images, a document repository, or an archive of voice mail recordings. These examples are drawn from real-world Nirvanix cloud storage customers, and it's hard to see where spot pricing fits in.

The typical cloud storage use case matches the graphic above, with steady growth of capacity over time. It's easy to see why cloud storage is so attractive, since capacity is provisioned and charged on demand rather than in large under-utilized steps. Regardless of whether internal systems are 90% or 10% over-provisioned, waste of storage capacity is a massive ROI opportunity for public cloud storage providers.

Some cloud storage users cap their applications, deleting old content as new data is added, but very few rapidly add and delete data. Cloud or not, storage just doesn't tend to fluctuate. One can imagine corner cases where capacity swings up and down, but these are not the norm.

One reason is the inertia of storage: It's physically difficult to move large amounts of data. Although cloud storage systems are remarkably open, allowing customers to easily migrate between providers, transmitting the data still takes time. The time and cost required to switch from one provider to another would likely wipe out any savings from spot pricing.

Getting Creative

Although spot pricing will likely never be widespread in storage, one can still imagine creative pricing strategies for cloud storage. Most providers, for example, now offer tiered pricing, discounting the per-GB cost as customers pass usage thresholds. Another popular concept is bundling expensive bandwidth to simplify the cost structure. Budgeted pricing is also popular, with a flat monthly invoice replacing the charge-as-you-go credit card model used by consumer-oriented services.

What might the future hold? Once can imagine further bundling and simplification, especially as standard APIs and applications are used to access cloud storage providers. Geographic arbitrage could soon become the norm, with remote or inexpensive data centers offering discount capacity. The federation of multiple providers is another exciting frontier to be explored. And some applications might even make use of spot pricing, but those are likely to be rare.

Everyone Should Be Skeptical About Cloud Service Providers

sfoskett — Thu, 22 Oct 2009 15:21:00 GMT

As we plummet down into Gartner's "trough of disillusionment", the cloud skeptics are making their voices heard. Although my professional focus is at the forefront of the cloud storage wave, I can not disagree with the content of articles with sensational headlines like "Cloud Storage: It's Strictly For Airheads" and "Why Cloud Storage Use Could Be Limited in Enterprises". The authors are doing exactly what everyone should be doing: Questioning the viability and suitability of cloud storage in the enterprise.

The truth is, although I'm not the "cloud police", not all managed storage services are created equal. In fact, lots of them are, to put it bluntly, not worth much. Many cloud backup and archiving services use bare un-protected disk drives to store data, have no redundancy built into the system, and try to scrape up every cent by using home-brewed hardware. This is especially true in the consumer space, where bargain-basement (or even free) pricing has driven a race to the bottom in terms of quality. No business should use junky consumer solutions.

Even service providers that presume to sell in the enterprise market often miss the mark. Forgetting the inappropriate per-month credit card billing method and laughably poor support services, many providers adamantly refuse to comply with basic corporate governance principles. Why would a business use a service that won't tell them where their data is, won't allow their auditors to examine both the processes and data centers in use, and won't stand up service level agreements (SLAs)? If your service is not enterprise-grade, you have no business selling to the enterprise! No wonder corporate folks are scared of the cloud.

Subpar offerings from flaky vendors hurt the whole industry. So let's take a look at what the business cloud storage skeptics are saying, and how the enterprise managed storage service providers can respond:

Data integrity is a must. Let's say you sold buckets for a living, but one out of 12 had holes in it. How long would you be in business? Amazingly, many cloud storage solutions lack both redundancy and integrity checks and use big bare disks to hold data. These buckets have holes, and data loss is a certainty. Fixing this issue is straightforward: All cloud storage should include parity protection, redundancy, and integrity checks. Business storage systems commonly use RAID to protect against disk media failures, but massive-scale cloud solutions must do even better. Cloud storage must also save multiple copies of data, preferrably automatically distributing it geographically according to SLAs. And I wouldn't be able to sleep at night without proactive integrity checking and data scrubbing.
Geographic dispersion of data has implications beyond data protection, though, so cloud providers must also include policy engines to drive governance and compliance. Since compliance with privacy and similar laws is based on physical location, businesses must know where their data is at all times. You can't have data growing legs and traveling, sometimes across national borders. In contrast to the old storage service providers (SSPs) of a decade ago, cloud storage systems are inherently policy-friendly. Objects, metadata, and policy-driven data management are becoming more common, enabling even better data governance than traditional enterprise storage systems. I bet your SAN or NAS system can not enforce your SLAs for geographic dispersion, keeping this data type in New Jersey and replication that other one to California but allowing only a third to cross to Europe.
Cloud storage services must be managed and operated better than enterprise systems, not worse. I cringed when I heard tales of the T-Mobile/Microsoft/Danger servers and SAN storage "sitting in a corner" with no knowledgeable staff, no operations focus, and a looming migration. Of course it failed! From an operations standpoint, I love that a humongous and homogenous storage infrastructure can be managed by just a few focused, educated, and talented individuals. That's what cloud storage is all about! Neglected systems have no business in production, especially at a service provider.
Managed services must allow auditors to verify their claims. I am a car nut, so I definitely wouldn't trust a garage who whisked my car off to an undisclosed location so unseen mechanics could work on it. I wouldn't eat at a restaurant that didn't allow the health department to inspect it. So why would I put blind faith in a managed service provider who held my critical data? Cloud vendors must perform their own security and operations audits and allow their customers to do the same. You can't pass the buck on governance: If you require SAS70 or PCI or a third-party audit, then your service providers must step up and allow it, too.

I'm mad, and I'm throwing down the gauntlet. I want every service provider to start now, protecting data, upholding policy, demonstrating operational excellence, and allowing audits. Anyone who doesn't is a disgrace to the industry, and their customers ought to seriously reconsider where they place their data. Mark me down as a cloud skeptic!

By the way, I work for a managed storage service/cloud storage provider, Nirvanix. But I'm a long-time enterprise storage consultant and would not work here if the company couldn't live up to these must-have requirements.

Cloud Storage Flavors: Platform/Infrastructure and Service/Product

sfoskett — Mon, 05 Oct 2009 15:45:00 GMT

Take a look at the various data storage offerings called "cloud" and your head will start to ache. How can so many things all bear the "cloud storage" name and yet be so totally different? The answer is obvious to long-time industry observers: Each provider has tailored their offering to make it distinct in the market, and each supports different use cases.

Infrastructure versus Platform

Although I am not one to tilt at windmills, especially when it comes to arguing the "rightness" of cloud-based marketing, I sometimes do hit upon a set of terminology that makes sense to me. Wading through the mess of "as-a-service" or "XaaS" names posted by my friend Greg Schulz the other day, it occurred to me that drawing a line between a platform and its underlying infrastructure is quite useful indeed. As Jeff Darcy points out, one can use this distinction to help make sense of the cloud storage landscape:

Cloud infrastructure resembles physical infrastructure in many technical ways. Consider Amazon's EBS or the 3PAR, NetApp, and new Symantec storage systems: They are highly scalable and can be used to support multi-tenancy in a service-based model but are also commonly used for very conventional purposes. Indeed, cloud cynics and skeptics would be forgiven for thinking of them as yesterday's fish wrapped in today's newspaper, since the technologies involved have changed very little from the pre-cloud era.
Cloud platforms resemble Internet applications and are thus much harder for infrastructure-focused folk to grasp. Offerings like Amazon S3 are a playground for programmers with REST-ish HTTP API access, but appliations expecting conventional blocks or files are left out in the cold. Cloud storage platforms are as different from SAN, NAS, and CAS as Google App Engine or Microsoft Azure is from VMware ESX or Citrix Xen.

When we encounter new beasts, we humans try to group them with more familiar things. Thus, a Pteropus becomes a flying fox, anime becomes a cartoon, and baseball is called rounders for adults. But these groupings often deceive us, hiding both the qualities and novelty of the item in question. Cloud compute and storage platforms are composed of conventional hardware, to be sure, but their uniqueness and value lies in how these building blocks are used.

Cloud storage platforms present a wholly new approach to data storage, with programmable access, integrated metadata, and the possibility of policy-based action. Last year, I called for a real revolution in storage, and today I recognize many of these elements being put into practice with cloud platforms. This is the reason I redirected my career and have devoted my time to evangelizing cloud systems in the enterprise IT community. Infrastructure has little strategic value any longer, so IT must move up the stack.

Service versus Product

Go-to-market approaches also vary among cloud storage products. Simply put, some are offered as services and others as products. Since the value proposition for cloud storage is so strongly rooted in the service provider concept (including multi-tenancy, scalability, high utilization, commodity hardware, and distribution of data), observers are often confused when they encounter a cloud product rather than a service.

Even the nomenclature can get confusing, with many referring to all cloud offerings as "services" regardless of the sales approach used. Many products are sold as "enabling cloud services" or "a foundation for SaaS", and the development of internal or hybrid cloud service providers is a hot topic in IT circles. Although many go-to-market approaches exist, it is fair to say that service-orientation is a hallmark of cloud products.

Enter the Cloud Storage Matrix

Applying these simple labels to the cloud storage market reveals a simple matrix of offerings. Some are clearly conventional SAN or NAS infrastructure products that can be leveraged as a foundation for cloud services. Others are true cloud platforms sold as a product for service providers. Then there is the storage capacity offered on a service basis by hosting providers, including the elastic block storage (EBS) included with Amazon's EC2 cloud compute infrastructure service. Finally, we have the true cloud storage platform services: Amazon S3, Nirvanix SDN, and Rackspace Cloud Files.

EMC's Atmos is an unusual beast indeed. It easily falls into all four quadrants, being sold as both a product and service and used as both a platform and infrastructure. Only EMC can tell if this "all things to all people" approach is working, but it will take some serious resources to make it a success. It certainly confounds the issue for those of us who are trying to bring clarity to the world of cloud storage!

Building Bridges

One thorny issue for cloud storage has been the unconventional HTTP-based access method required by true cloud platforms. This has been something of a chicken-and-egg quandry for businesses developing cloud storage offerings: Conventional block and file protocols enable accessibility and promote usage but also limit the impact of the platform. As Steve Duplessie points out, cloud storage offerings must include real strategic value rather than being sold simply as cheap rental space.

One way to build a bridge between today's world of SAN and NAS and the future of programmable storage platforms is to include both access methods. This is the reason that Nirvanix rolled out its bridge to the SDN, CloudNAS, last year. Conventional applications can read and write to the Nirvanix cloud storage service as if they were accessing a local file system. CloudNAS handles the translation between POSIX filesystem calls on a UNIX or Windows server and RESTful HTTP calls to the cloud. It also includes caching and encryption to address the two most common concerns about cloud storage, performance and security.

Although conventional access methods and product offerings offer an onramp to the cloud, the real revolution will not come until applications are able to seamlessly access stored objects. This is why cross-platform programming libraries like the Zend Simple Cloud API and CloudLoop for Java are so interesting: They bridge a new breed of applications to the cloud storage platforms already offered. Cloud storage is moving up and to the right, adopting both the platform and service orientations that are the hallmarks of the Internet age.

Why Isn't Storage Getting Cheaper? Part 5: Storage as a Service

sfoskett — Fri, 18 Sep 2009 01:18:00 GMT

Although the capacity of storage systems keeps growing, data growth keeps absorbing available capacity. In an effort to contain costs, IT organizations turned to tiered storage but have encountered a "glass floor": Disk space is a small component of costs, reducing the impact of hardware solutions. IT organizations have long turned to outsourced services to enable cost savings on non-core infrastructure and organizational elements, and manaed storage as a service promises to deliver remarkable savings.

Why isn't storage getting cheaper? This series of articles attempts to answer this question:

Storage as a Service

Managed services are not a new idea, and savvy IT organizations have been benefitting from their application for many years. Managed storage as a service has existed for over a decade, although the realities of available bandwidth and latency have limited its use. Where it has been applied, storage as a service has proved an effective cost-savings mechanism, and has provided other benefits as well.

One aspect of storage as a service that is often overlooked is the operational management included with enterprise-grade services. Storage and backup service providers providing on-demand services must take on many of the tasks associated with (and adding cost to) enterprise storage:

They monitor capacity and utilization and report on the growth of storage
They monitor availability of the system and the reliability of its components, adjusting and repairing as needed
They design and engineer the storage system and handle hardware and software procurement duties

These are serious time-savers in their own right, and become must-haves when one is considering cloud storage. Gearing up an enterprise storage team to select, engineer, roll out, and manage a novel concept like object-based cloud storage would require easily two to three times the effort of implementing a new NAS or SAN system. It isn't that cloud storage is especially difficult (it's actually pretty straightforward) but that it's unlike anything used in enterprise IT before. Gone are conventional ideas of RAID, LUNs, SAN and NAS protocols, and the rest of the old-school storage baggage. In its place are storage and database servers leveraging commodity hardware and lots and lots of disks.

Turning to a managed service provider instead just makes sense. Companies like Nirvanix specialize in managing this kind of environment, focusing all of their energy on it. Although the archival or lower-tier data one often finds on cloud platforms might not be the most precious asset, a cloud provider with an SLA will treat it like royalty. But the best part is the purchasing model: Sign up and pay for only what you use. Larger environments can arrange flat all-inclusive "budgeted" billing, but there is no need to waste too much time on capacity planning since capacity can always expand on demand.

Considering the all-inclusive nature and usage-only billing of fully-managed cloud storage, it's easy to see why TCO comparisons are so surprising. Storage service providers are not working magic: By offering highly-utilized and standardized storage, they can cut storage costs to the bone. Storage is indeed getting cheaper, and organizations that turn to the cloud are realizing the savings.