More virtual machines than physical machines were sold last year. What does that mean for storage?

As noted 4 years ago in The virtual machine I/O blender

Engineers have spent decades optimizing the OS, drivers, caching, controllers and disks for specific workloads.

Observed behavior such as locality of reference have informed many strategies. Like read-ahead.


But when you put 25 virtual machines on a single server, what happens to all this hard-won empiricism? It’s gone.

The new empiricism
Enter Tintri. Their mission: making storage for virtual machines more efficient.

Co-founder Kieran Harty was an engineering VP at VMware. The other co-founder, Mark Gritter, was part of Kealia, a server company founded by Andy Bechtolsheim and bought by Sun. Architect Ed Lee has deep storage experience dating from Berkeley’s RAID work in the late 80s.

Tintri brings several ideas to the issue of VM storage.

• A VM-aware file system
• Inline dedup, compression and a hybrid file system that manages flash and SATA drives as a single pool
• Managing VMs directly through interaction with vCenter
• Manage VMs, not LUNS or volumes
• Automatic VM alignment

The StorageMojo take
Much of this mirrors good ideas that other 21st century storage companies have adopted. The big win is the Tintri’s direct management of VMs.

What this means in practice is that an admin looks directly at the I/O activity of each VM, not on a LUN or volume basis, but at the VM level. Which means the array is doing the same thing, able to optimize performance for each VM based on what the VM is doing in real time.

If a VM starts gobbling up storage and bandwidth, it’s easy to see which VM is the problem. This should be tied to a good chargeback system as well.

There are many products that improve – often dramatically – VM storage and performance. But making the VM the unit of storage is brilliant and opens the door to many more enhancements.

Tintri is expecting to support other hypervisors in the not-too-distant future.

Courteous comments welcome, of course. I attended the Tintri presentation as a Tech Field Day delegate. The vendors who presented at the TFD paid for the privilege, which funded my travel expenses.

Over the years StorageMojo has seen several architectures that just seemed smart, but whose market potential was blighted by management and funding issues. Violin Memory was one. Coraid was another.

In both cases a new CEO – at Coraid, Kevin Brown – has made a world of difference. New funding, new customers, more focus. All good.

What does Coraid do?
Storage, of course. Inexpensive block-based, scale-out, network storage. Global namespace.

Define inexpensive
How does $0.40/GB sound?

How?
Let’s start at the server. The HBAs are Intel NICs that are re-programmed to present themselves as SCSI controllers. The server sees a SCSI controller, but it’s Ethernet out the other side.

Running over the network is a connectionless datagram protocol that accesses the storage on commodity x86 servers. The protocol is fast and lightweight, being based on the simple and well-understood ATA command set.

There’s a router for connecting sites, but most users use it in a single site.

Who?
People who want a lot of inexpensive block storage. Average order is ≈175 TB. Average sales cycle is 30 days.

Coraid is a horizontal play, but they get interest from the usual big data suspects: media and entertainment; science; service providers; private clouds.

The StorageMojo take
Coraid takes the commodity game seriously: commodity servers; commodity HBAs; commodity open-source protocol; commodity network; and commodity SCSI drivers. Not much custom engineering required.

It’s prices reflect that. There’s a huge unmet need for fast, cheap storage that Coraid is tapping. Their biggest problem is that they do things differently than everyone else, and it takes people a while to decide to work with those differences.

Which is a shame. The more competition for the block storage dollar the better it is for all consumers.

Courteous comments welcome, of course. I saw Coraid as part of a Tech Field Day program, where they, among other vendors, helped pay for transportation, food and lodging.

Spoke to Chris Gladwin, founder and CEO of Cleversafe at NAB 2012. Cleversafe had stopped communicating a few years ago – usually a bad sign – so an update was long overdue.

When last heard from, Cleversafe had an ISP/MSP target market, offered an open-source version of their software, and focused on safely archiving confidential data on public networks. No more.

A 50PB order
CEOs are professional optimists. But Chris’s story was good.

Their 1st order was 100TB. The 2nd, 50 petabytes. That is a lot of boxes to rack.

Now Cleversafe focuses on multi-petabyte orders. They can handle 5-7 such orders a year.

Patents
But when they aren’t installing petabytes of disk, they’re writing patents. Hundreds of them.

Chris thought they were up to 268 patent applications. The USPTO shows 21 granted patents, including 7,904,475 Virtualized data storage vaults on a dispersed data storage network, 7,853,710 Methods and devices for controlling the rate of a pull protocol, 7,844,712 Hybrid open-loop and closed-loop erasure-coded fragment retrieval process, 7,818,518 System for rebuilding dispersed data,” 7,818,430, Methods and systems for fast segment reconstruction, and 7,574,579 Metadata management system for an information dispersed storage system along with another 181 applications yet to be granted.

At a conservative $25k per patent in legal and filing fees and lost engineering time, that’s $6.7 million. If there’s another startup as aggressive on patents, I haven’t heard of it.

The StorageMojo take
It’s seems that most, if not all, of Cleversafe’s business comes from the US intelligence community, not commercial users. Otherwise we’d see reference sites and more interest from top-tier VCs.

Regardless, Cleversafe’s strategy of massive orders, massive patents and limited fulfillment is unlike any other in the industry. Is the limited fulfillment due to a complex product – the GPFS of scale-out storage – or a limited market?

As many HPC-focused firms have found, it can be difficult to shift from extremely specialized high-end government customers to commercial users. Companies that have, like Panasas, have had to work to keep their products general purpose, avoiding the honey-trap of fascinating but one-off designs.

Cleversafe’s pivot from the commercial market and open-source may reflect a 1st mover disadvantage: too early to the commercial market, they’ve been co-opted by the government market. But the bigger concern is whether or not that massive patent portfolio will stall development of better high-scale storage systems.

Cleversafe’s exit strategy seems to have almost as much to do with patents as it does with building a business. Are they the first storage company hoping for a buyout by a patent troll?

Courteous comments welcome, of course. I’ve recently done work for Panasas and am working with a company – Amplidata – being sued by Cleversafe.

Maximizing value for time spent is a difficult calculus. Especially when it the goal is perspective rather than information.

Efficient use of your time will be why the Solid State Storage Symposium works. Starting at 10am at the Doubletree San Jose Airport and going until 4pm, you’ll get to hear about – and question – 9 different SSS-using vendors.

The crack StorageMojo research team will be there in force, delivering a keynote that looks at the next 10 years in storage technology and architecture. Not everyone will agree with every point – that would be no fun at all – but there is no doubt that the next 10 years will see more change than the last 30.

This is a tech-heavy symposium with many of the smartest people in leading edge architectures speaking, including Dave Wright of SolidFire, Suresh Vasudevan of Nimble Storage, Jonathan Goldick of Violin Memory and Jered Floyd of Permabit among others. The panels intend to elicit critical differences, not dry recitations of marketing hype.

It should be good.

The StorageMojo take
One of the reasons big companies dominate is that it is so hard for small companies to get their unique value across to users. This is an opportunity for the up-and-comers to explain what it is they do, and for tech-savvy users to judge for themselves the goodness these vendors are offering.

Hope to see you there!

Courteous comments welcome, of course. I’ve done work for Nimble and Violin.

Our digital civilization requires data integrity and long-term preservation, and neither is assured by our current storage infrastructure. But progress continues.

Latest case in point: Amplidata. This 4 year old company, based in Belgium with a growing US footprint, brings a new level of erasure code goodness to the both problems with a cluster-based object store.

What erasure code goodness, you ask? The first – AFAIK – rateless erasure code, AKA fountain code, storage system in production use.

And that is a good thing because?
Robustness and efficiency.

Amplidata claims storage durability well beyond RAID 6: 10 9′s (spread across 16 drives with up to 4 failures) durability – though the spreads can be much larger logically and geographically. They do this by breaking the data object into segments and adding redundancy data.

The redundancy data adds about 50% to the object size – more efficient than mirroring or triple replication. The benefit is that the system can lose hundreds of segments and still reconstruct the data.

Each object is protected by checksums that can protect against more than 1000 simultaneous bit errors per object. And each write goes to at least to controllers before it is committed.

What kind of monster controller is able to perform all this magic? The minimum configuration is 3 Xeon-based commodity controller nodes with as many 10-drive Atom-based storage nodes as you need.

Amplidata is optimized for bandwidth, not IOPS. With their latest software update they now spec each controller at 750MB/sec, and you can have as many controllers as you can afford.

Sounds like Cleversafe
Cleversafe thought so too, and they’ve sued Amplidata for patent infringement. But Intel – who knows about patents and due diligence – invested after the suit.

Like NetApp’s suit against ZFS, this seems like a vanity project. Surely Cleversafe has more important things to invest in. If they don’t they’re in bigger trouble than we know.

The StorageMojo take
The need for robust, inexpensive and massive storage has been a theme of StorageMojo’s for years. Object storage is the best solution to the problem of scale, while the kind of redundancy and end-to-end checksumming that Amplidata uses seems as robust as anything on the market today.

As for inexpensive, that is in the eye of the beholder, but Amplidata tells me that their newest storage node lists for less than $0.60/GB while consuming only 60 watts. That should be attractive to people running tape silos who want faster access and better redundancy.

Courteous comments welcome, of course. I’m working with Amplidata to produce a video white paper on their technology, so stay tuned for more info on a promising company.

Over 3 years ago StorageMojo saw that Violin Memory was “. . . on the winning architectural track.” Well, it took a lot of time and money, but Violin is making good on that early promise.

StorageMojo’s enthusiasm was kindled by Violin’s unique architecture. Here’s a short video that shows how Violin’s architecture addresses key problems with flash:

Full screen mode recommended.

The StorageMojo take
The industry is still in the early days of digesting the implications of fast persistent solid state storage. We’ve built up 50 years of cruft to deal with disk’s many issues. It will take a few more years for flash’s new options to ripple through the entire storage, server and application stack.

Take, for example, failover. If all apps and monitoring software could declare a failure in 10 seconds rather than, say, a minute, how much smoother would major apps run? How much better would be the perception of system uptime and response times be?

There are many other possibilities – what about metadata? – that flash and its successor technologies will affect. I’ll be offering more detail in my keynote at the Solid State Storage Symposium on Wednesday, April 25 in Silicon Valley. S4 is free and you can register here.

Courteous comments welcome, of course. The other flash company I liked in 2009 was Fusion-io, and they’ve done OK. And yes, Violin paid StorageMojo to produce the video white paper, but the opinions are my own.

Actually, StorageMojo is wrong, wrong, wrong, wrong and wrong. Umesh Maheshwari of Nimble Storage wrote a detailed and thoughtful response to the StorageMojo post Are SSD-based arrays a bad idea?

The StorageMojo take
Umesh makes good points, but perhaps due to Nimble’s hybrid disk/SSD architecture some seem to miss the mark. What’s missing is ample consideration of what the alternative to an SSD might be, a problem Nimble didn’t have because SSDs work fine, technically and economically, for their hybrid system.

For example, on the issue of reliability Umesh says:

Reliability. There are good reasons to be able to replace failed flash devices similar to how hard disks can be hot swapped. The raw bit error rate (RBER) of flash is actually worse than that of hard disks, and it gets worse as blocks are rewritten. It is also getting worse as manufacturers are moving to increase density. (See this paper from FAST 2012: The Bleak Future of NAND Flash and a related blog post.)

This is correct, but based on the Google/Bianca Schroeder research, the StorageMojo point is that the disk electronics – apart from the head/media pieces – are a major – 40%-50% – source of HDD/SSD failures. The flash controller has to handle the RBER and declining flash performance, but why add the other HDD bits that account for a substantial percentage of drive failures?

I could niggle about Umesh’s other points, but what fun is that? StorageMojo readers are encouraged to check out Umesh’s post and make up their own minds.

Courteous comments welcome, of course. I recently did a nifty video white paper for Nimble, which is a great intro to their innovative architecture. Check it out.

More is coming on SSDs RSN, but in the meantime there is the following piece from Virsto’s Eric Burgener on HA considerations for SSDs. Virsto is a software company focused on making VIRtual STOrage for VMware and HyperV much more functional than the physical kind.

Thus Eric’s response has a very particular POV: what is needed to use SSDs in a virtual environment to ensure high availability – from a company that DOESN’T sell HA hardware. One key point: virtual server environments are much more write intensive than most enterprise apps, so using SSDs as a cache is a losing strategy.

If that is intriguing, read on!

In Thinking About SSD, You Can’t Leave HA Considerations Out

In the “host vs array-based SSD” discussion as it pertains to enterprise accounts, the need for HA must play a critical role. This is true whether you’re working with physical or virtual environments. Any committed data that is not sitting on a shared, non-volatile, external storage device and accessible by at least one other node cannot be recovered until that failed node (on which it resides locally) is brought back up.

There are technical ways to solve this using synchronous replication technologies, but that’s an extra credit project you do yourself for now – as of yet, that hasn’t been built into any host-based SSD products. The reality today is that using host-based SSD precludes the use of HA (but not necessarily things like vMotion, which is NOT HA).

This was touched on in some other posts, but I think it’s an increasingly critical issue in virtual computing environments that may have been a bit downplayed in other comments. If you’re either thinking about moving production server workloads to VMs or have already got them there, HA is critical for a high percentage of workloads.

I can’t imagine an enterprise customer spec’ing out a production virtual server environment without asking about HA. True, there are workloads that don’t require it, but most do.

And it’s not just virtual server environments. We’re running into an increasing number of VDI environments where they want to enable HA for at least a small percentage of the desktops – usually executive desktops. HA isn’t a deal breaker for VDI like it is for “VSI” (virtual server infrastructure), but there are clearly use cases in VDI where you want and/or need it.

Today SSD is pretty much only used as a cache, regardless of where its deployed. And to provide a given level of performance speedup, caches generally have been sized at somewhere around 2% – 4% of the primary data store (it varies by application and exactly what you’re trying to speed up).

In virtual environments, write performance is much more critical because it tends to comprise a much higher percentage of the read/write workload – in VSI environments its not uncommon to see 50% reads/50% writes, and in VDI environments we’ve seen 70% write environments. Unless you’re using a write back cache (with all the attendant additional expense associated with that), you’re not going to get any write performance speedup from the conventional cache architectures, just read.

But now think about what a log architecture, applied at the storage layer, could add. Circular logs that are continuously draining (asynchronously) as they are filling need very little storage capacity to speed up ALL writes for ALL VMs ALL the time. In our experience, you need a log of about 10GB in size for each heavily loaded physical host.

Think about what that could mean for a 16 host environment with 20TB. You could get away with 2-4 200GB enterprise flash drives instead of the 10-12 that you might otherwise deploy in a 20TB environment. If you have a “linked clone” type snapshot technology combined with storage tiering, you could take the extra SSD capacity and create a tier 0 for critical VMs that need very high read performance, like for example the golden masters in a VDI environment or common templates you use to create your server VMs.

This covers both needs – read and write performance – using a lot less storage. That means pretty much the same performance you’d get with the more expensive configuration with more SSDs for a lot less money. If you want to use SSD efficiently, a log architecture is a great idea. And if the logs are placed in shared, non-volatile, external storage (like the SSDs hosted in a SAN array or SAN-based SSD appliance), you can fully support HA.

Host-based SSD cards are closer to the physical host so theoretically they’ll provide more performance speedup, but given Amdahl’s law, how much of that can you really use? Array-based SSD will still get you past storage latencies as your critical bottleneck, and if they’re implemented using a log based architecture you’ll get HA and large write performance speedups as well using a lot less of it.

The StorageMojo take
Write-through caches avoid a lot of sticky update synchronization problems, but as Eric notes they aren’t the best choice in write-intensive environments. And HA adds to the requirements: the cache must be network accessible.

But his larger point bears repeating: SSDs are wonderful for handling metadata. And as we move to object storage and more metadata that capability will become even more valuable.

Courteous comments welcome, of course. I think Virsto’s architecture is smart and fixes some real problems with VMware and vMotion.

StorageMojo offered its soapbox to any vendors willing to weigh in on the question of whether enterprise arrays should be built from flash SSDs or not. Ed Lee, architect at Tintri, formerly of Data Domain and a Berkeley Ph.D, elected to respond. It is a long piece but rich in insight.

Tintri produces hybrid disk/flash SSD appliances optimized for virtual environments, not Symm-killers. They use SSDs in their products, as do other folks like Nimble Storage.

No money changed hands between Tintri and StorageMojo or related entities. My accountant is weeping in the next room.

Begin Tintri’s response:

Outside the SSD Box: More than Faster Disk
Robin Harris of Storage Mojo in his recent article, “Are SSD-based arrays a bad idea? and Matt Kixmoeller of Pure in his response, The SSD is Key to Economic Flash Arrays, present interesting perspectives on whether or not SSDs are the best technology for building flash-based arrays. Robin argues that by rethinking how flash can be packaged outside the SSD box, you can achieve better performance, reliability, cost and flexibility. And these observations are supported by the experience of existing flash-based storage vendors who have developed their own custom flash modules and packaging. Matt argues that SSDs provide an industry-standard product that requires less investment to leverage, better economies of scale, and rapid improvement in technology. These are also very valid points, especially for startups with limited time and capital.

Latency
Taking latency as a point for comparison, flash-based storage vendors using custom packaging often quote IO latencies in the tens of microseconds versus SSD latencies of low hundreds of microseconds. While this is a notable difference, software and interfaces can also add overhead and the final latency seen at the subsystem level may differ by only a factor of two to four. Server-side flash products can avoid more of the software and interface overhead and provide better latencies – but may require rewriting applications to capitalize on this advantage. Keep in mind that hard disk latencies can easily reach tens of milliseconds under even moderate load. ALL of these flash-based products have latencies that are hundreds of times faster than disk.

In short, most of the performance improvement comes from simply replacing hard disk with some form of flash. This immediately shifts the performance bottleneck from storage to some other component in your system. As a result, you won’t be able to take full advantage of flash performance without also optimizing the performance of the rest of your infrastructure, and ultimately rewriting your applications as well.

The above phenomenon explains why replacing your hard disk with flash often speeds up your applications by only a factor of two to three rather than ten or a hundred. Congratulations! You’ve just moved the bottleneck from storage to some other component of your system. By Amdahl’s Law, further improving only storage performance has diminishing returns. So while custom packaging does provide significant advantages in latency, most applications are unlikely to benefit until the rest of the computing ecosystem is optimized to take full advantage of flash.

To take a closer look at SSD latencies, I ran the following simple experiment:
1) Erase an MLC SSD so that no logical blocks were actually mapped to flash, and then issue small random reads.
2) Overwrite the entire SSD so that all logical blocks are mapped, and issue the same small random reads in step 1.

The idea here is to measure the software and protocol overheads of accessing flash packaged as SSD separately from accessing the data on the SSD. Reads with no blocks mapped had latencies of around 70us, while the reads with all blocks mapped had latencies of 250us. In this case only a fraction of the overall IO latency was due to SW and protocol overhead, indicating that SSDs may still have significant room for improving latency.

Form factor
Another important issue discussed by both Robin and Matt is the relative cost of flash packaged in SSD versus non-SSD form factors. Robin argues that an SSD costs significantly more $/GB than the underlying flash while Matt argues that non-SSD packaging is expensive to develop, and SSDs provide useful flash management functions as well as hot-swap capability. It’s certainly true that developing custom packaging has a high up front cost, although this is likely balanced by lower unit costs. But as Robin points out, there are also standard packaging options available for non-SSD form factor flash, which may make custom packaging for non-SSD flash unnecessary.

A very important point to keep in mind when thinking about commercially available SSD vs. non-SSD form factors is that SSDs are designed as a substitute for disk, while non-SSD form factors are often designed as substitutes for memory. This means that SSDs focus primarily on reducing $/GB (its greatest weakness vs. disk), while non-SSDs focus on reducing $/IOPS (its greatest weakness vs. DRAM). This explains why SSD is currently much cheaper on a $/GB basis than PCIe flash, while PCIe flash designed as memory expansion is cheaper on a $/IOPS basis than SSD. This is not to say that you can’t build a non-SSD form factor that has lower $/GB than SSD, just that the primary applications for these non-SSD form factors today is usually not as a replacement for disk.

Whether flash in SSD versus non-SSD form factors is better for use in storage subsystems in the long run primarily depends on the relative volumes of these products, and the feature and price sensitivity of the applications these products serve. At this point the ‘winning’ form-factor seems hard to predict. So as a flash subsystem vendor, it seems desirable to keep your options open and ensure that your technology will work well with a variety of packaging options.

More than just a faster disk
But flash is about more than just performance and packing. Flash enables much more than just a faster, denser replacement for disk. With flash, we can finally remove a key mechanical barrier to scaling not only storage systems, but computing systems in general. Going forward, CPU, network and storage can now all scale with improvements in semiconductor technology. When transistors replaced vacuum tubes, we got more than just compact radios; we got simpler, more powerful computing systems. Similarly, flash is a catalyst that will enable far greater levels of automation and functionality for storage and computing systems than is possible today.

I tend to think of the value of new technology as the product of its simplicity times the functionality it offers. It’s clear why functionality is important, but why is simplicity so important? Technology that is simple to use will be used more often, to solve more problems, in less time. As a result, simplicity has a compounding effect on value:

Value = Simplicity * Functionality

How does one measure simplicity? One way is to list the basic steps it takes to perform a task and how long each step takes. One to three is good, four to six is manageable, and anything resembling a twelve step program will likely require written directions and a significant amount of focus. Note that in assessing the simplicity and functionality of a technology, one must do it in the context of the job that needs to be done. For example, a chainsaw has great features for cutting down trees but not for giving haircuts.

A common problem with many general purpose storage products when applied to applications such as virtualization is that they require executing long lists of steps to get anything done – and most of the features are not directly applicable to virtualization. Paradoxically, many of the features that try to make these products better suited to the application end up making the products more complex – resulting in little improvement in overall value. Kind of like adding too many tools to a Swiss army knife until you have so many that the attachments start to stick and rub against each other.

Flash as a catalyst
Flash eliminates a key mechanical barrier to scaling computing systems and is 400 times faster than disk. To keep things in perspective, the speed of sound is “only” 250 times faster than walking! If I could get to work at supersonic speeds, I would no doubt save a lot of time each year. But would I do no more which such an ability? Similarly, is flash just a faster replacement for disk? Will it make no significant difference in the way storage is managed and used? We obviously don’t think so. Flash will greatly increase the value of storage by improving both the simplicity and functionality of enterprise storage products. But these gains will not come easily or without their own set of problems.

An obvious way flash promotes simplicity is by eliminating performance bottlenecks, but as flash enables more dense storage systems many of those gains will be converted to problems in quality-of-service. A more significant way flash promotes value is by providing a better building block for constructing storage systems: flash promotes simplicity by enabling higher levels of automation and allows the implementation of more powerful functionality.

Flash will fragment the enterprise storage market. The general purpose storage systems of today will be supplanted by new flash-based products that are far simpler and more powerful for the specific application areas that they target. This will amplify the simplicity and power that flash already makes possible, and further accelerate the fragmentation of the storage market. This is precisely what happened in the 1980’s when advances in networking technology caused a shift from centralized computing to networked computing – and in the process fragmented the direct attached storage market into ones based on networked storage technology. Over time, the networked storage markets consolidated into the current general purpose storage market dominated by a few major vendors. And so the cycle is repeating itself.

We are at the start of a new technological shift. A shift that is made possible by flash and one that will disrupt the existing enterprise storage market. Just as transistors enabled new products such as personal computers and smart phones, flash will enable simple, intelligent and fast enterprise storage systems. In turn, this will lead to much higher value for end users, but only if we think outside the storage box and treat flash as more than just a faster, denser disk.

The StorageMojo take
For the record the original post wasn’t looking at hybrid solutions, although it is obvious that SSDs can help legacy designs stay competitive without replacing all disks for a few years. For folks like Tintri and Nimble who want to speed up disk storage to stay affordable SSDs make sense. Why engineer a small part of your system when an off-the-shelf solution will suffice?

But for high end transactional SAN storage I still don’t see how SSDs are the right way to go. But I’m expecting more responses, so stay tuned.

Courteous comments welcome, of course. I’m working on a post that reflects directly on Ed’s comment about SSD latency. You’ll like it.

A reader asks:

Do you have any tool to move External Files (nearly 70 TB) from Celerra & Centera to Isilon faster?

The StorageMojo take
I know EMC made it difficult to leave their Centera system for competitive systems, but making it difficult to leave for another EMC product seems perverse. Or maybe they don’t know how to do it either.

Readers, or Isiloners, any suggestions?

Courteous comments welcome, of course.

Pure’s Matt Kixmoeller saw the Are SSD-based arrays a bad idea post and, unsurprisingly, responded. The SSD is Key to Economic Flash Arrays is a good post and I urge interested readers to check it out.

Pure has a stellar team with deep experience. Their views are worth considering.

As Matt notes:

This post caught our eye for an obvious reason: Pure Storage did start “fresh” to build an all-flash enterprise storage array, and we did decide to use the SSD form factor, after quite exhaustive looks at all the other options. Quite simply, we found that SSDs are the most efficient and economic building blocks from which to build a flash array. Let’s explore why.

After dismissing disk arrays that add flash drives – as I do – Matt focuses on (1) all flash appliances built from raw NAND and (2) flash arrays using flash SSDs.

SSDs are most efficient
Matt argues that SSD-based arrays have 3 key advantages:

• Economics. SSDs are a commodity product that raw flash arrays will have a hard time out-engineering.
• Flash controller complexity. Matt notes, correctly, that the flash controller is at the heart of argument. Better to use a controller that goes into millions of SSDs or one purpose-built for a single vendor’s array? How will the single vendor be able to keep up?
• Servicability. Pure’s use of SSDs enables them to offer a familiar hot-swap experience that higher density designs may not offer. Futhermore, Pure’s data reduction features increase effective density to rival raw flash designs.

In conclusion, Matt makes a couple of more points. First, that SSD form factors will become much more compact, such as Apple’s DIMM-like mini-SATA SSD used in the MacBook Air. Second, that the proof is in the pudding: Pure, he says, has “. . . delivered with break-through performance, at a cost below traditional spinning disk.”

The StorageMojo take
How does Matt’s response stack up to the criteria in the original post? Not that there’s anything magic about them, but . . . .

• Latency. No response, which doesn’t mean they’re worse.
• SSD bandwidth. No response, but to be fair with enough SSDs you should be able saturate 16Gb Fibre Channel.
• Reliability. No direct response. Instead a focus on servicability. More on that below.
• Cost. Says Pure is cost-effective using their data reduction technology.
• Flexibility. This is the heart of Matt’s argument: due to the commodity volume of the flash controllers flash SSDs will evolve faster – in functionality and cost – than any proprietary solution could. Proprietary flash controllers, he says, will be boat anchors for flash array vendors and are likely to end up controlled by flash manufacturers.

Servicability is an interesting response to the question of reliability. After all, the reason hot swap is important for some components but not others is because they either a)fail often – individually or in aggregate – b)failure compromises the product or c)online expansion, upgrading or reconfiguation is desirable.

Power supplies are routinely hot swappable because they have the lowest MTBF of any major system component. Disks are hot swappable because they come in multiples that reduce their aggregate MTBF while their standardized design makes hot swap cheap. I/O cards are often hot swappable because they are critical and needs change.

SSDs should be hot swappable because their failure rates are at best about half that of disks. But DIMMs, another critical component, especially if you invest in high-capacity ones, aren’t, because they rarely fail.

While I’m not aware of any non-SSD enterprise array vendor whose arrays don’t include hot swap components – love to be educated – which is more important: a short mean time to repair (MTTR) or a long mean time between failures (MTBF)? Because that is the argument about servicability.

I’d like to publish responses from vendors who feel strongly about this issue. Not in the comments, but as a blog post. Any takers?

Courteous comments welcome, of course. I was so impressed with the Pure Storage team that I signed a rare NDA with them last spring to get briefed, the first of 2 visits to their Castro street HQ.

The friendly folks at Panasas are sponsoring Taming the Big Data Beast: Big Data for Design and Discovery at 10am PDT. I’ll present the StorageMojo take on big data.

I’d like to hear from you on any issues I should address. Feel free to comment or email me at robin at this domain.

Update: Here’s the link to the WMV file for the webinar.

The StorageMojo take
Panasas founder Garth Gibson – he of the original Berkeley RAID paper – was so far in advance of the rest of the industry with scale-out architecture, object storage and extreme bandwidth that it is only in the last few years that enterprises have caught on to why these are all Good Things. I’m glad they’ve hung in there and pleased by their support for StorageMojo.

Courteous comments welcome, of course. I’ll be in Silicon Valley Wednesday morning with some free time. I’d like to see cool stuff that people are working on.

Think: if NAND flash storage arrays were being developed today, what is the chance that we’d put the flash into little bricks and then plug a bunch of them into a backplane? So why do it now?

It is a truism of design that when a new technology is developed, we use it to build what we have today. It is only in later generations that we realize the new possibilities enabled by the technology. And those generations can be long, even in computers.

For all out talk about the rapid pace of computer innovation, the market for the tried-and-true is much larger than the one innovators fight over.

Why SSD-based arrays are a bad idea
To be clear, this discussion covers storage arrays built with standards-based (i.e. SATA, SAS, 2.5″ or similar) SSDs.

• Latency. Low compared to disks, but substantial compared to flash. SAS/SATA stacks were never optimized because disk latency was the big problem.
• SSD bandwidth. There are wider options, especially close to the CPU.
• Reliability. SSDs replace the head/media assembly in disk drives with NAND chips. The rest of the SSD has all the tender bits of a regular disk – bits that account for about half of all disk failures. Compare DIMM and disk replacement rates.
• Cost. SSDs cost 50%-100% more than the raw flash, even after using all the high-volume disk components. Mounting directly on PC boards, like DIMMs or PCIe cards, is much more cost effective.
• Flexibility. The good news with SSDs is that they take advantage of the huge tech infrastructure that supports disks. But that’s the bad news too, if an optimized clean-sheet architecture is the goal.

How big an issue is cost? DRAM on a DIMM is ≈98% of the DIMM’s cost, where the flash in an SSD ≈50%-65% of the cost. And since flash costs are dropping faster than the other component costs, so will its percentage of SSD cost.

Given the high cost of flash media compared to disk, efficient media usage is a major issue. Will flash SSDs pass that test?

A less important but related metric: rackspace. SSDs are inefficient users of racks, taking perhaps 2x the space of non-SSD flash arrays per TB. Few customers will care, but the ones who do write big checks.

The StorageMojo take
The massive technological momentum behind SSD-based arrays make them a popular option for both vendors and customers. After 20 years of RAID arrays, customers get the model. There’s a large raft of hardware and software support for disk drives that SSDs can use.

That cuts time-to-market and development cost. Given the performance advantages of SSDs over disks it is an easy win for customers even if the architecture is sub-optimal.

The squeeze comes later: if non-SSD architectures have significant advantages the SSD-based arrays will lose market share and gross margin. Flash-based SSDs make sense for many applications where their cost is a small percentage of the total solution.

Building storage arrays from SSDs is opportunistic, not strategic. It isn’t the future for high-end storage, but less-demanding mid-markets may not care.

Courteous comments welcome, of course. I’m really interested in any holes in the logic of this analysis. Please weigh in.

Another StorageMojo Best paper, The Bleak Future of NAND Flash Memory, presented at this year’s FAST ’12 conference, quantifies flash’s declining reliability, endurance, and performance as density increases.

Researchers Laura M. Grupp and Steven Swanson from the UCSD Non-volatile Systems Lab and John D. Davis of Microsoft Research collected data from 45 flash chips from 6 manufacturers. Using that empirical data they predict the performance and cost characteristics of future SSDs.

Faster better cheaper or slower worse cheaper?
While NAND flash is produced with semiconductor processes, smaller feature sizes don’t lead to faster performance or greater reliability. As NAND features shrink, so do the number of trapped electrons that store information.

Figures of merit
The research found that performance, program/erase endurance, energy efficiency, and data retention time all got worse with feature shrink.

Based on past performance, the team derived equations to describe how changes in feature size have affected key specs. They looked at SLC, MLC and TLC and feature sizes scaled from 72 nm to 6.5 nm (the consensus smallest feature size published in the International Technology Roadmap for Semiconductors (ITRS0), and assumed a fixed silicon budget for flash storage.

Key results

• Latency. MLC write latency will double over time. Triple-level cell writes will grow to over 2.5MS, noticably reducing its performance advantage over disk writes.
• Bandwidth. Small – 512B – read bandwidth and all writes decline by up to 50% over time. The impact is greatest on high-performance SLC flash.
• IOPS. MLC flash I/O rates will drop almost in half.

Flash may be the new disk in a few years.

The StorageMojo take
One important qualifier is that for the purposes of their modeling the team constrained the number of chips in the hypothetical future devices whose performance they predicted. While fine for isolating the impact of future chip shrinks, it ignores the potential of much greater parallelism for managing these changes.

Bandwidth drops by half? Double the number of chips.

But if something can’t go on forever, it won’t. NAND flash will soon enter an end-of-life crisis for computer applications that need performance. That’s why ReRAM (resistance RAM) looks to be a good bet for replacing computer flash – not mobile device flash – over the next decade.

Courteous comments welcome, of course. A version of this post was published on ZDNet last week.