This focus means that it’s become rather noisy in the market with claims of superiority and the finding of the flash “Holy Grail” ringing out near constantly.  This in turn leads to performance claims where benchmarks are contorted in every conceivable way possible to eke out new claims of performance leadership.  All of this noise can make it harder on a customer to separate fact from fiction and truly understand what approach can best help them address their current needs and future directions.  Much like the scene in Indiana Jones and the Last Crusade where there are a number of possible “Grails” in front of Dr. Jones, there are just as many flash options.

As my colleague Hu Yoshida noted last November with the introduction of Hitachi Accelerated Flash storage, HDS has been looking for ways to take flash to higher levels of performance, durability, capacity and cost, which leads to improvements in overall productivity.  We ship many petabytes of flash capacity each quarter to our customers.  With that in mind, we want to share a few thoughts on where flash is, and where flash is going. Since there are too many flash topics to cover in one blog post, this is the first in a series of flash-focused blog topics which will help in choosing flash the right way today.

Let’s start with the basics.  Given all the chatter, customers need to be able to answer the following questions for themselves when considering a flash purchase.

• Does the provider offer a suite of flash capabilities (tiered systems, all flash systems, server flash, no flash, etc.) or will they attempt to convince you that their approach is the only one that works?
• Will I need to change my management schema to deal with a new flash storage system or will it work within the tools I already have (or might add) from this vendor?
• Can I get an all-flash version of this vendor’s technology? Alternatively, if I buy a high-performance, all-flash array today, can I add disk drives and tiering capabilities later if my needs change?
• Can I add both block and file storage to the flash system?  Does it offer unified management across those environments?
• Will I get the same level of enterprise availability with flash as I get with my traditional hard-drive based storage?  (e.g., non-disruptive upgrades, predictive sparing, etc.)
• Since the applications that need high-performance tend to be those that need the best protection, can I get built-in replication?  If so, do I need to learn new replication tools for flash or will my tools work seamlessly with my disk-based storage systems?
• How will this vendor’s technology help me most efficiently use my flash investment – does it focus on de-duplication, compression or tiering to drive down costs?  (Can I use these technologies in secure, encrypted environments?)
• Can I virtualize my current storage assets behind my new flash storage system to extend the useful life of those assets and accelerate their performance?
• Do I trust this vendor will be focused on this same technology direction 12 months from now?  Will this vendor even be around 18 months from now?
• Does my storage warranty ensure that I am covered for any solid-state memory issues despite how active my storage system is?
• Can the vendor articulate a strategy of tiering across the infrastructure and flash deployment options?
• Can this vendor’s flash strategy help me across all my environments – including virtualized servers and mainframes?
• How does a vendor protect and service the solid state memory within its systems?  If a chip fails, is my array in need of swap or can I swap an SSD/flash module?

You might be wondering if this was a self-serving list of questions.  Maybe a bit. But even so, it is a relevant list of more than just a few of the questions many of our customers are asking.

I can also assure you that somewhere in that list there’s at least one question that will make every vendor a bit uneasy, including HDS.  Also, I’m sure I missed some relevant questions too, and I’d be interested to hear which you think should have made the list. I’ll be sharing insight into each of these questions in a few upcoming blog posts.  And many we’ll cover off on in upcoming announcements, so stay tuned.

Hopefully a list like this can help customers cut through some of the slick marketing slides and get to the facts about the choices available in the market.  In keeping with the Grail Knight’s counsel, “choose wisely, for while the true Grail will bring you life, the false Grail will take it from you.”

Many technology marketers like to draw an analogy between data in the IT world and blood in the human body. It makes sense because, if you lose your data / blood, or if it stops flowing, the organization / body will likely die.

This analogy, however, fails to recognize one important difference between IT and nature: the human body has a fantastic ability to regulate the amount of blood it contains; as new blood cells are created, old ones die. The alternative would be reminiscent of several horror and sci-fi movies you may have seen.

It doesn’t work that way in IT, though; new data is created much faster than older data is expired, deleted or destroyed. Consider this: storage capacity in personal computers of the 1980s was expressed in kilobytes; in the 1990s we were talking about megabytes; then gigabytes in the 2000s; and now many of us have at least a terabyte on our desks, and we’re not even half way into the 2010s. That’s roughly a 1000x increase per decade. Does anyone really doubt that we’ll need petabytes of capacity by 2020 for our computer, tablet, smartphone, wristwatch, glasses, car, cloud, or whatever comes after cloud?

Of course, much of this new data is created from new sources (online transactions, social networking, smart devices, sensors, etc.) and from improvements in older sources, such higher resolution scientific equipment, video and imaging. But most of the new data is really just copies of the other data. We have backup copies for operational recovery, off-site copies for disaster recovery, copies for development & test, copies for long term archive, copies for off-line analysis, and on and on. We’re drowning in the copies of our data.

The data storage and management vendor community has been developing technologies to try to contain this growth – such as data compression, data deduplication, automated archive and deletion processes, snapshots, etc.  for years. But as we continue to see the amount of data grow we’ll need better solutions, especially for managing and limiting the number of copies of data. Have you started to hear the word ‘exabytes’ yet?

That’s why, as we launch a new solution category around data instance management, beginning with the recent announcement of Hitachi Data Instance Manager, the timing is perfect for me to join HDS.

My job as senior product marketing manager is to help build the data protection and instance management story that shows how the implementation of this new capability will result in exceptional business outcomes for our customers. The journey, for me has just begun and I hope you’ll come along for the ride. Follow me on Twitter @rcvining.

About the author: Rich Vining has worked in the secondary storage industry for more than 20 years, helping to shape many of the solutions that protect and retain data. The contents of this blog are his own, and do not necessarily reflect the positions of Hitachi Data Systems or any

As storage infrastructures continue to grow in size and in complexity, significant challenges arise to properly monitor storage capacity growth and performance service levels delivered to critical business applications.  Since storage systems are often viewed with a level of unknown within the IT environment, i.e. “black box”, storage is often the first to be blamed when an unexplained application performance issue arises. This often prompts storage administrators to scramble for detailed storage performance statistics in an attempt to prove storage is not at fault. Typically, this involves manual tracking processes that may not be completely accurate nor properly scaled in an enterprise environment. In addition, as enterprises are constantly evolving, manually maintaining and monitoring the pertinent storage (supply) to business application (consumer) relationships can be taxing.  Enter storage analytics, where storage service level agreements and objectives that include detailed measurements of storage capacity and performance can be defined and measured for key business applications. With each business application having differing service level needs for storage, ensuring each of your critical business applications are meeting their required storage service levels is a challenging task that storage analytics can help solve.

We recently sat down with one of our software product manager at Hitachi Data Systems, Manu Batra, to find out more about storage analytics and what Hitachi Data Systems is doing in this area.

What is a service level agreement (SLA) and service level objective (SLO) for storage?

Manu: In storage terms, I would define a SLA as the service levels my business applications are expecting from the storage resources they rely upon.  SLOs are specific metrics, such as response times/latencies or utilized storage capacity, which together define the storage SLA for a particular business application.

What is the best way to measure the success of a storage SLA?

Manu: Essentially, a SLA would use both response time and data throughput metrics to verify that the storage infrastructure is delivering the required service levels to an application at any time. This is how you would typically define a SLA from the storage layer to any layer above it (server, application, etc.).

Now that we have defined a SLA and SLO for storage, it seems that they used to be utilized more in the service provider industry.  So how is this relevant to an enterprise; could they use the same metrics as well?

Manu: As enterprises become larger and data centers grow beyond their original designs, it’s more complex for any enterprise to keep a comprehensive view across its entire infrastructure.  They need quantifiable measurements that determine how their data center is doing today.  Instead of looking at all aspects of the data center independently, time can be saved by looking at a set of key storage metrics that can be set, managed and optimized.  SLOs provide customers a standard set of performance and capacity metrics that allows enterprises of all sizes to quantify a ‘Level of Happiness’ from their storage infrastructure.

What are some of the benefits enterprises receive by applying storage analytics to their infrastructure?

Manu:  As mentioned earlier, a lot of time is spent analyzing just to ensure storage is not at fault for an application performance issue. By applying storage SLOs to key business applications, you are better equipped to quickly respond to application performance issues, maintain storage service levels for critical business applications, and you can simplify storage troubleshooting. Even in today’s growing enterprise storage environment, IT staff resources are limited and they need to continue focusing on saving money with their current infrastructure. So the question they must ask is “How can we maximize our existing storage resources as the data center’s capacity expands?”  In addition, storage analytics provide greater visibility to CIOs, architects and storage administrators, allowing them to continuously check on the status of their infrastructure with a storage dashboard, get notified early on for any potential issues, and accurately measure storage capacity growth over time.

How does Hitachi Data Systems deliver storage analytics to its customers?

Manu: Hitachi Data Systems is deeply committed to helping customers manage and optimize their IT infrastructure.  HDS has been delivering storage analytics capabilities for some time as well as continuous research focused on helping organizations manage the tremendous data growth in their data center.  Hitachi Command Director software enables the concept of SLAs and SLOs to be applied to storage and helps customers adapt to ensure storage service level agreements for business applications are being met.  It enables the data center to be properly measured beyond an operational view to a business-centric view to increase storage utilization and efficiency in order to better address critical business application needs.

On May 9^th 2013, I attended the Cloud2020 Summit at SuperNAP. One of the hotly debated discussions was “Programmable Infrastructure”.  (Check out Pete Johnson’s blog to get a glimpse of what was discussed).  When I asked how far they think the infrastructure has come in terms of programmability, Randy Bias, co-founder & CTO at Cloudscaling, responded that the network is a mess, and storage is probably second worst. Imagine the kind of innovation we could see with the infrastructure platforms, if they were as programmable as applications are. As I ponder that, I keep coming back to RESTful APIs. I really believe that RESTful APIs are key in enabling the innovation around infrastructure.

In my previous post I highlighted two key characteristics for Intelligent Platforms that:

• can expand or shrink with the demand fluctuation
• can be programmed

In this blog I’ll explain the role of RESTful, by taking a storage use case.

Imagine you have an instance of an application with X amount of allocated storage.

Say the storage demand for this application increased and you need an incremental storage of Y amount needed – otherwise your application would crash. Would you let the application crash and risk the consequences? Wouldn’t it be nice if your infrastructure was intelligent and could:

•  monitor the storage utilization of the application
• set the storage thresholds and trigger an alarm when the utilization reaches that threshold
• request the underlying storage array to expand the capacity

This should be simple, but unfortunately it depends.  It’s not that simple today because of the underlying components that make up the stack. In a simplified view, you have the application, VM, hypervisor, network, and the storage array.  Different vendors provide different components of that stack.  This is where the RESTful APIs become relevant.

At the core of the RESTful API principles  you will find:

• the application of engineering—generality to the component interfaces
• the stateless nature, that allows each interaction to be independent of the others
• the ability to introduce intermediary components to perform different operations like load balancing, security enforcement, or encapsulating legacy systems

We kept these aspects close to our design when we built the UCP Director. In the rest of this blog, I will explain how to automate the above use case and the role of RESTful API. More information about the specifics of UCP for VMware vSphere can be found here.

Because UCP Pro for VMware vSphere is built for VMware stack, the above use case can be accomplished by essentially bringing two sets of subsystems together – VMware vCenter and UCP Director.

Read here for  instructions on how to accomplish the VMware side of the operations.

Here is how the RESTful API looks from UCP Director side. All you need to do is to make a POST call to UCP Director and you are good to go.

Request header nicely breaks down the resources – first it provides identifier for the storage subsystems, followed by the identifier for the volume itself that you want to expand. The body of the request has the size of the new volume, after expanding the original volume. Just like the body of the request, the body of response is also represented in JSON.  This makes it easy to consume the response in a programmatic way. In the response, you will notice an entry called “JobId”. Using this JobId you can track the status of the job.

Beauty of the RESTful API from UCP Director is that tomorrow the exact same API can be consumed by non-VMware infrastructure. Just like Jennie found a way to block TV spoilers on her twitter feeds, RESTful API’s generality nature and its ability to introduce intermediary components enables customers to innovate around their the IT infrastructure to meet their unique needs.

As part of the Information Forums in Mainz and London, I’ll be moderating panel discussions on big data with HDS executives, which should be both entertaining and enlightening. We will cut through the hype and discuss practical big data applications today while taking a look to the future and how the scale of data retention will evolve to create unique business value.

By attending, you will gain meaningful insights and tactical tips for evolving your datacenter to gain the most efficiency and reduce expenses. In addition, you will also be among the first to hear about innovative solutions and strategies such as:

Maximize Your IT Advantage – Discover how to accelerate performance, control your data growth, and transform your IT environment to be more intelligent, efficient, and cloud-ready.

Cloud, Your Way – Learn how to improve agility and flexibility through various private, public and hybrid cloud solutions provided by HDS and our partners.

Capitalize on Big Data – See examples of innovative, big data solutions today that transform how you capture and capitalize on data.

Click here to register for the HDS Information Forum in Mainz, Germany

Click here to register for the HDS Information Forum in London, UK

But corporate entities at branch offices also require storage to get their work done.  Their IT needs will typically be smaller, but their data needs will also continue to grow.  To mitigate this situation, Hitachi Data Ingestor can be installed at these locations as a cache for frequently used data, making it locally available with the vast majority of the data stored at the core HCP and accessed via links.

We complete our content cloud with the incorporation of the new HCP Anywhere (announced today), our enterprise-ready, safe, secure file synchronization and sharing solution.  HCP Anywhere is an enabler to mobile and remote employees, allowing them to collaborate and share files efficiently and securely, eliminating the need for them to use consumer clouds that can compromise security.

Finally, the picture is made complete by offering customers the means to offload data and content to a hosted cloud with offsite hosted cloud solutions that allow customers to reduce or eliminate additional capital costs and better control operational costs.

This tightly integrated portfolio, coupled with our new delivery model and partner programs enables our customers to deploy their own clouds, in their own ways allowing them to focus on how to take advantage of the transformation from traditional IT to cloud rather than trying to figure out how to get from where they are now to where they are going next.

An exciting new solution is Hitachi Content Platform (HCP) Anywhere-  the industry’s 1st integrated file sync and share designed for the enterprise and built by a single, well respected vendor. This integration-by-design puts HCP Anywhere ahead of the competition in security, ease of implementation, keeping IT in control. HCP Anywhere is:

• Secure: HCP Anywhere is an on-premise, private cloud solution that keeps data within the control and governance of corporate IT.
• Simple: Users download the HCP Anywhere client application from a portal or via the iTunes store, self-register their devices, and they are ready to go since it works just like other folders on their device.
• Smart: HCP Anywhere uses the backup, compression, single instancing, spin-down disk support, and metadata capabilities of HCP to create a highly efficient content sharing environment.

A new version of Hitachi Content Platform (HCP) plays an important and foundational role in the HDS cloud portfolio.  With this announcement, HDS expands  the robust capabilities of HCP as a cloud storage platform, featuring the most advanced metadata management in the market today.   A more feature-rich metadata capability means HCP can better help automate operations,  providing a strong foundation for a big data repository and helping find the right datasets for deeper analytics.

Also featured are new hosted cloud services from HDS.  These include Hitachi Cloud Services, an extension of the enterprise into a secure, robust off-premise hosted cloud managed by HDS, and the Hitachi Cloud Service Provider Program, a partner provided public cloud offering built on HDS cloud infrastructure and solutions.

With today’s announcement, HDS delivers another milestone on its strategy for the cloud with solutions that first and foremost addresses what customers need.  Our new and enhanced enabling technologies and cloud solutions help you:

• Choose the best possible solutions to address your needs, at your own pace, and in a way that makes sense for your business
• Solve the challenges you face today – and gives you a variety of new options to help manage the explosive growth of unstructured content and achieve the cost and flexibility benefits of the cloud by reducing CAPEX and OPEX
• Leverage cloud and file sharing technologies in ways that were never before possible. With these new offerings, you can be prepared for what’s coming next. (big data, bring your own device, next-generation file services, secure clouds, distributed IT, metadata-driven automation, management and analytics)

This of course leaves them with the following alternatives:

1. Work with a provider who can manage a private cloud that resides within the customer’s data center, built on assets owned by the provider
2. Choose a public cloud that charges for services delivered based on usage, allocates resources on demand, and supports a service level agreement (SLA) consistent with the needs of the customer

So how does one decide? Because an organization’s data and content are critical to deriving insight used to make sound business decisions, they are strategic assets to be seen and used only by those authorized to do so. For this reason, any cloud approach must ensure adequate data protection. At the same time, access to data and content, while carefully controlled, must be unfettered to those who need to find, use, and share it. Then effective collaboration and innovation can take place. Finally, the cloud must support the policies and processes needed to meet whatever compliance and regulatory requirements that apply to the data involved.

It goes without saying that performance and reliability are also important considerations. In the case of private clouds, the robustness of the hardware, software, and networking elements that make up the cloud environment are fundamental to determining if these criteria are met. This is true for public clouds as well, of course, although in this case they are reflected at a higher level in the commitment the provider makes through its SLA.

One could argue that there is something approximating a tradeoff between reaping the cost and resource benefits of the cloud and meeting security, access control, performance, and reliability requirements. The more that is owned and/or managed by a provider, the more control one could perceive they are relinquishing. That introduces risk – or, at the very least, the perception of risk.

The solution to this, of course, is to work with a provider that can assume the role of a “trusted advisor.” One that not only possesses the experience and knowledge to build, host, and manage quality cloud solutions, but one that has demonstrated the ability to deliver them securely. And one that is willing to back its performance and security claims with a strong SLA. The foundation of whatever they host has to be a set of proven platforms and technologies that can meet these demands.

In other words –put your strategic data in the hands of a provider who can prove they are worthy of it.

At the heart of this growing problem are data retention policies, which haven’t evolved much over the last 10 years. While we have more dynamic backup and archive (or data management) software, the cost of storage hasn’t dropped as quickly as growing consumption. Chris mentioned it wasn’t uncommon for his clients to have 60 to 100% annual data growth which is fueling concerns about the cost of data retention. Martin’s post specifically speaks to the total cost of a petabyte-sized tape archive. While the upfront cost of tape media is attractive, the realistic cost and time of maintaining a tape archive when you consider media migration and management is daunting. From Martin’s post:

It currently takes 88 days to migrate a petabyte of data from LTO5-to-LTO6; this assumes 24×7, no drive issues, no media issues and a pair of drives to migrate the data. You will also be loading about 500 tapes and unloading about 500 tapes. You can cut this time by putting in more drives but your costs will soon start escalate as SAN ports, servers and periphery infrastructure mounts up.

Further complicating this equation for CIOs is the promise of big data analytics and the challenge of extracting business value out of this retained data. As a standalone archive, tape doesn’t inherently lend itself to frequent access and data mining.

This challenge is larger than the medium used to store data. Today, data retention policies are often developed around retention costs and legal compliance and less on strategic business value. Copy management of backup and disaster recovery data also continues to compound the problem as does growing machine-to-machine (M2M) data. For many companies, petabyte-sized data retention is becoming the norm, not the exception.

Hitachi is no stranger to this challenge. As a $96 Billion dollar global conglomerate with business units that include Information and Telecommunication Systems, Electronic Systems and Equipment, Social Infrastructure and Industrial Systems, Automotive Systems,  Construction Machinery and Financial Services…this challenge is personal.

All of this points to the need for data retention architectures which span hardware, software and people.

Metadata and Object Storage

As more data is generated, cost effective, accessible long term data retention is going to be dependent on the optimization of storage resources and good utilization of storage metadata. Greg Schulz summed this up recently in FedTech:

For some applications, an important attribute of storage solutions and services are their metadata capabilities. This includes the ability to support flexible and user-defined metadata.

Another enabling capability is policy management, which can use metadata for implementing or driving functions such as how long to retain data, when and how to securely dispose of it, and where to keep it (along with application-related information). This adds some flexible structure to unstructured data without the limits or constraints associated with structured data management.

This is where object storage can play a significant role. By leveraging application metadata that describes the content and its data retention policies, it can enable automation of those policies, freeing up storage and human resources, as well as managing the problem of unnecessary duplication of data.

Storage Tiering

While the concept of storage tiering has been around awhile, I’m surprised how infrequently it is used in many data centers. If implemented properly, storage tiering provides dramatic optimization of high performance (and more expensive) types of storage while intelligently leveraging lower cost storage based on performance, capacity and reliability characteristics. Both structured and unstructured data can be highly optimized in most environments. Database applications, in particular, can benefit significantly from tiering. Frequently referenced data, such as control tables and indexes can reside on the highest performance storage tier while less frequently accessed reference pages can be moved to a lower tier of storage. I believe we’ll see a much broader adoption of automatic (or dynamic) tiering as budgets force IT administrators to gain better storage utilization rates.

Primary Storage Deduplication

Data deduplication has been broadly adopted for backup data, but adoption of primary storage deduplication has not. One of the biggest barriers to primary storage deduplication is the very real concern of performance degradation. Like so many storage technologies, primary storage deduplication has evolved to have little or no impact to performance. Like tiering, deduplication processes can be automated. Administrators no longer need to experiment with scheduling and tuning deduplication processes.

Conclusion

These aren’t the only technologies which can be used to optimize storage utilization, but they should be among the options you consider as you look at managing your growing data requirements. The challenge of large data is very real and data retention policies will need to evolve as well. Intelligent data management, which includes developing compliant data expiration practices, will also need to develop as a matter of discipline.

• can expand or shrink with the demand fluctuation,
• can heal on its own,
• can be programmed,
• is available 24/7 anywhere, any device

You’re probably seeing an overarching theme across HDS around enabling our next generation platforms to intelligently react with information in the platforms. As our CEO Jack puts it we Innovate with Information.  Software Defined Data Center, Cloud Infrastructure, Cloud Management are some of the current aspects of the IT industry that help make these intelligent platforms real.

Hitachi Data Systems has always innovated and contributed to society in a sustainable way.  Our recent launch of Hitachi UCP solutions is a great example of this effort. The work done with Hitachi UCP Pro for VMware vSphere not only allows our customers to leverage their investments in VMware infrastructures, but also arms them with REST API to programmatically talk to UCP. Look for a subsequent post from me on this topic.

As we innovate and enable customers to be ready for their future needs, we value the role our technology and channel partners play and we continue to work with them as we add more intelligence into our platforms. A great example for this is the recent launch of UCP Select with Cisco UCS servers, bringing Cisco’s server technology to our customers and empower our channel to add additional value.

From an IT platform point of view there are clearly some major movements happening in the industry– such as OpenStack, flash technologies, unified storage, unified fabric, and big data, and HDS will continue to make contributions in all of those areas. I have the privilege of shaping some of these efforts by playing the new role as CTO of Intelligent Platforms. I am honored and humbled by this new role and I hope to use this blog as a forum to engage with you all and to bring some technology insights into various efforts as we evolve. Speaking of OpenStack – I’ll also be writing about some of the near term activities we are doing in this area. Stay tuned and until then, here is another perspective on the Intelligent Platforms from our good friend Krishnan Subramanian.