Today, Amazon’s Werner Vogels announced the Amazon Relational Database Service (RDS), which is a fully MySQL 5.1-compatible database as a service offering.  Sure, Joyent launched their MySQL Accelerator in August, but nobody seemed to notice.  Today’s RDS announcement from Amazon, combined with SQL Azure, heralds a new era in production scale RDBMS-as-a-Service (RDBaaS).  By answering SQL Azure and fully validating the RDBaaS opportunity, Amazon has jump-started a fundamental transition in the $18B market for database systems.

DBMSs are now part of the utility model of cloud computing.

Historically, DBMSs have been something that you licensed (CapEx), installed, and physically managed and administered.  Going forward, DBMSs will be something you use and perform logical administration and tuning on top of — but you won’t install, configure them, or worry about your log files jamming your disks and crashing your Web site.  The RDBMS will just be there, ready for your applications, automatically backed up and replicated, and operated flawlessly as part of the fabric of your cloud.

The DBMS Market Dynamic

By providing MySQL, Amazon is catering to the Web and SaaS crowd, and less so to the enterprise.  Conversely, Microsoft is well-positioned to compete in the far larger and more lucrative enterprise RDBMS market.  Yes, there is MySQL in the enterprise, but it’s a side-show to Oracle, SQL Server and IBM’s DB2 (with a sprinkling of Sybase and others).

Speaking of which, how will Oracle, IBM and Sybase respond?  IBM has their cloud offerings and will support DB2 fully – but will they be as innovative?  Oracle’s stalled acquisition of Sun may eventually lead to an Oracle cloud where they would be able to offer a similar service.  And frankly, out of all of the DBMSs out there, Oracle’s users have the most to gain from not having to hassle with that big and hard to manage system.  Sybase?  They seem to be dateless at this point.  They don’t offer a cloud (and likely won’t), can’t get a leading cloud to back them with their small market share, and tend to only be used in really intense applications like trading and risk analytics on Wall Street – which are less likely to migrate to the cloud soon.

I don’t see a scenario where MySQL-based DBaaS offerings aren’t dominating the Web/SaaS market (MySQL the software dominates it today).  Nor do I see a market for enterprise DBaaS that doesn’t have SQL Azure in the lead.   There will be crossover (more with MySQL in the enterprise than SQL Azure in the Web/SaaS environment).  Oracle may be able to craft a solution that enterprise-focused cloud providers (Terremark, Unisys, etc.) can use, but I wouldn’t bet on it soon.

Amazon RDS and Microsoft SQL Azure Compared

This is not intended to be a rigorous review, but there are some fairly significant differences in the approach taken by these vendors that are worth exploring.

Deployment Model

SQL Azure is a multi-tenant service with multiple DBs on the same machine in a shared infrastructure environment.  In some ways this is similar to how I believe Rackspace manages MySQL in their Cloud Sites business.  You don’t manage the types of instance you run other than to select from two pricing tiers.  The pros of this model is that they can offer SQL Server for as low as $10/month up to 1GB (or $99/month for 10G — nothing in the middle).  If there’s a downside, it’s that some users might want to feel more like they can kick the dbms more directly.  Also, by having a 10GB limit they force design decisions on applications with larger databases. They recommend partitioning (sharding) your data set across  multiple DBMS instances to manage more than 10GB.

RDS is a bit different in that they provision individual special-purpose EC2 instances into your AWS account with a larger range of databases — from 5GB to 1 terabyte!  They also let you control the type of instance, from Small (1.7GB RAM, 1 virtual core, etc) to Quadruple Extra Large – no foam – (68GB RAM, 8 souped up virtual cores, etc.).  This gives you more control, but also makes you have to think more about your database — but then that’s how AWS works overall.  You can get pretty large (1TB) without sharding due to size, though you may have to shard your DBMS for performance as this is not a “scale out” solution.  My friends at Akiba Technologies in Boston are building a killer engine that eliminates sharding and radically improves performance for most MySQL applications. They’re a couple months from alpha, but this stuff is seriously kickin’ and should be considered by any other cloud provider feeling the need to compete in the RDBaaS space (disclosure, I am an advisor to Akiba).

One note is that, unlike SQL Azure, your bill can get pretty large for RDS.  The smallest instance, is $0.11/hour, which is $80.30 per month (they are planning “reserved instance” pricing in the future – which will reduce that price).  For their largest server, the bill is $2,270/month – plus you pay for the DBMS storage separately at $0.10/GB).

Database Support

SQL Azure is mostly compatible with SQL Server, but not 100%.  SQL Azure supports a reasonably large subset of T-SQL.  Here is their FAQ on compatibility:

SQL Azure is built on SQL Server database technologies that are used for running mission-critical applications in the enterprise as well as on the Web. Since SQL Server is a broad data platform that can handle all data types from birth to archival, there are many associated capabilities that the data platform provides. SQL Azure is exposing a large subset of these relational capabilities and extending them as services in the cloud.

These services feature built-in high scale, availability, and self-management, and are provided in a way that makes it easy for customers and partners to consume over the Internet. SQL Azure, in its first iteration, exposes only the core RDBMS capabilities of what is in the full SQL Server data platform.

RDS, conversely, is 100% compatible with MySQL 5.1.  This is pretty sweet!

There are many more things to cover, and perhaps I’ll continue this post at another time.  The bottom line is that the market now has two credible, robust and innovative RDBaaS solutions, and it won’t be long before there are more.  Welcome to the era of the DBMS Utility.

Note – I make the (internal) distinction for a reason.  The term “private cloud” is now starting to morph from purely internal, to internal and external clouds controlled closely by IT.  Amazon’s Virtual Private Cloud is an example of a private cloud in an external provider setting.

I have seen charts from Gartner that show how private (internal) clouds will get more money from IT over the next few years than public clouds.  I’ve also seen the benefits of a private cloud in the development/test workload scenario here at Unisys.  The numbers are pretty staggering (we are publishing a paper on this).

I can imagine that IT folks are just a wee bit confused by all the cloud noise lately, especially when it comes to private clouds.  You can choose from a wide range of approaches.  These start with open source cloud projects from Eucalyptus or OpenNebula providing a good base level cloud framework.  Then there’s the guys who have been out there a bit but are not really all that big — like Enomaly or 3tera (both of whom seem to see greener fields selling cloud solutions to hosting and telco providers than to entperprises).

In the 800-lb gorilla category you have VMware with their vCloud initiative.  Not that most VMware customers aren’t already locked in until Armageddon, but vCloud is just one more way to remove any possibility in the future of getting out from under VMware’s thumb.  vCloud=vLock! (note – as always on these pages, this is me talking, not my employer)

There’s also the big iron guys like IBM with CloudBurst and HP CloudAssure (is that even a product?).  In this collection you can go from free (open source) to over $200,000 (HP, IBM) just to open the box. Oh, and it normally comes in a box, as in appliance (or appliance with cloud capacity in a rack).

So, what does a private internal cloud solution look like from an IT perspective anyway?

Well, it looks a bit like how Amazon’s EC2 and S3 look  to Amazon’s internal tech crew.  It starts with a bunch of hardware that can run the workloads.  To this you add a level of virtualization and/or grid control to enable multiple workloads to run on the same box.  This can be Xen (as in Amazon), or it can be VMware ESXi or Hyper-V, etc.  Then you add layers of automation that allow you to manage tens, hundreds, thousands or even tens of thousands of boxes with relatively few people.  My friend John Willis (e.g. @botchagalupe) calls this “zero touch” infrastructure (note, John is working with the Canonical folks on the Ubuntu Cloud bundle with Eucalyptus).   “Zero touch” covers server repurposing, virtualization, image remediation (patch and release level management), provisioning, metering (for chargebacks) and more.  Then you add self-service interfaces (portal, API, etc.) and voila!   Private cloud.  Note – I’m pretty sure that Eucalyptus and OpenNebula fall far short of this level of automation out of the box.

As you might imagine, this is bloody difficult to pull off.  Amazon, Google and others worked their keisters off to get to where they are, and they are still building.  Basically, the primary activity of the data center folks at Amazon is swapping servers (and that’s a big job).  One fails, another is swapped in, and the system grabs the new box and joins it to the cloud.  In most IT organizations, a lot of people get involved in installing the image on the machine, virtualizing it, and joining it to the network.  For obvious reasons, that wouldn’t scale at Amazon.

One of the first things IT needs to let go of, to get a cloud running to its full potential, is the concept of manually approving anything.  All of the rules for approvals need to exist in advance and the system just needs to follow them.  Having a person “OK” the creation of a VM is just crazy, yet that’s what happens in many IT organizations.  Someone requests a virtual machine instance for a project and it goes into a queue for manual approval, and then manual implementation.  Yikes!!  This isn’t to say that nothing is approved – as I said, you have to set up the rules in advance.  How many instances of X can a department get?  What do we charge them (IT is now really a business, no?)?  Are there time limits on the VMs or servers we provide (e.g. maximum lease terms)?  All of this needs to be there and ready to go.

Then you need to give people the portal. That’s the magic pixie dust that makes it all come together.  Rather than “requesting” a server, you provision one.  It might take a few minutes for the automati0n to do its work, but that’s a lot better than the hours, days, or even weeks (yes weeks for a single VM!!) it used to take.

So, to net it out — a private (internal) cloud = hardware + virtualization + automation + chargeback + self-service portal. It also = the promised land for enterprise IT modernization.

Sounds easy?  As if!

The explosion in distributed computing brought on by ever faster processors (coupled by lower prices on CPUs, memory and storage) allowed for the client/server revolution to push workloads onto the client and off of the server.  Today, much of the compute power of edge devices (PCs, laptops and smart phones) is not used for computing, but for presentation.  Raw workload processing is happening on the server to an increasing degree.

Much like a supernova that collapses into a black hole, workload distribution is becoming ever more concentrated in the data center.  But in this case the data center is also being concentrated. The rise of the hosting economy that started with the internet era continues to accelerate.  But the level of concentration of data center resources into a small number of super galactic providers is accelerating even faster.  Amazon and Google are gaining share while much of the traditional hosting and outsourcing community is losing out.  Some may not see it yet because their sales are still increasing, but a short look at some of the analysis by Guy Rosen on Amazon vs. Rackspace, GoGrid and Joyent shows that even some of the larger players out there are losing share to the galactic few.

This concentration is affecting the corporate data center too, of course.  More than half of Amazon’s instances are going to large corporate users according to CTO Werner Vogels.  It’s still very early days, and most large companies continue to invest in data center infrastructure, but you don’t have to be a genius to see that at some point the cloud concentration effect will be felt by even the largest of IT organizations.  No matter how you measure it, cloud computing is going to have a material impact on the number of servers and related components that the average enterprise is responsible for managing.  With the possible exception of storage, the effect will be to shrink the corporate data center.

So what does this have to do with IT staffing?  The galactic-scale cloud providers have invested heavily in a level of automation that most enterprise IT organizations can only dream of.  The result is that the number of servers per systems administrator at Amazon or Google is likely many times (if not orders of magnitude) larger than your typical IT shop.  As more and more of the data center moves to the cloud and consolidates in the galactic core, the need for systems administrators and related IT staff functions will be diminished.

Until the cloud, Moore’s law resulted in a steady increase in demand for skilled systems and network administrators.  At some point, the economies of scale and concentrating effects of cloud computing – particularly in the area of IT operations – will be visible as a measurable decline in the demand for these skills.

This reminds me of a t-shirt I got from 3tera at Interop in May, the back of which reads “Real men don’t manage servers (anymore).”

The opinions expressed on this blog and in related Twitter, Facebook or other feeds are mine alone and do not reflect the opinions of my employer – Unisys – or anyone else for that matter including but not limited to current, former or future clients, dogs and cats in my neighborhood, or the toad that keeps finding its way onto my porch (ftw?).  Okay?  Okay.

Open source was a software movement that began as an idea and now dominates how many new software offerings are marketed and sold. Open source is not a technology, but a business and legal framework within which technology is propogated.  Still, the biggest companies in software are largely closed source – Oracle, SAP, etc.  Nearly all specialty vertical apps (e.g. trading systems) are closed source.  Whereas most new development technologies including databases and tools are open source.  Given that open source is more a legal construct which bleeds into sales and marketing, it’s highly likely that there will be both open and closed source models co-existing for any foreseeable future.  Further, open source shrinks the size of the industry from a revenue perspective by default  (though parodoxically, software spending is up this year even in this economy).

What about cloud computing?  Will there still be the need for the cloud modifier in the future?  In the past most infrastructure was sold directly to the users under a capex spending model – this includes servers, databases, operating systems, etc.  Of total infrastructure spending in 20 years, how much will still be for on-premise capex, and how much for cloud opex?  Will ecconomies of scale drive infrastructure in the cloud to a point where the infrastructure market will shrink in both real and nominal terms?

Will the purveyors of servers, networking and core infrastructure software sell 90% of their wares to cloud companies?  Will what we currently call  ”cloud computing” be  just plain “computing” in the future?  Time will tell, and it will be a long time before the cloud distinction becomes superflous, but it will be interesting to watch.

“Private Cloud is not the Cloud

These CIOs know that what is sometimes dubbed “private cloud” does not meet their goal as it does not give them the benefits of the cloud: true elasticity and capex elimination. Virtualization and increased automation may give them some improvements in utilization, but they would still be holding the capital, and the operational cost would still be significantly higher.”

What if we were to look at the private cloud concept as an interoperability play?  If someone implements a cloud-like automation, provisioning and management infrastructure in their data center to gain many of the internal business process benefits of cloud computing (perhaps without the financial benefits of opex vs. capex and elastic “up/down scaling”), it still can be a very valuable component of a cloud computing strategy.    It’s not “the Cloud” as Werner points out.  It’s just part of the cloud.

To realize this benefit requires a certain degree of interoperability and integration between my “fixed asset cloud” and the public “variable expense cloud” such that I can use and manage them as a single elastic cloud (this is what is meant by “hybrid cloud”).  Remember that enterprises will always need some non-zero, non-trivial level of computing resources to run their business.  It is possible that these assets can be acquired and operated over a 3-5 year window at a lower TCO than public cloud equivalents (in terms of compute and storage resource).

Managing these fixed + variable hybrid cloud environments in an interoperable way requires tools such as cloud brokers (RightScale, CloudKick, CloudSwitch, etc.).  It also requires your internal cloud management layer to be compatible with these tools.  Enterprise outsourcers like Terremark, Unisys and others may also provide uniform environments for their clients to operate in this hybrid world. In hybrid you get the benefits of full elasticity since your view of the data center includes the public cloud providers you have enabled. You may choose to stop all new capex going forward while leveraging the value of prior capex (sunk costs) you’ve already made.  In this context, private cloud is very much part of your cloud computing strategy.  A purely walled off private cloud with no public cloud interoperability is really not a cloud computing strategy – on this point I agree with Vogels.

Co-Generation:  Selling Your Private Cloud By the Drink

Now, assuming you’ve built a really great data center capability, implemented a full hybrid cloud environment with interoperability and great security, what’s to stop you from turning around and selling off any excess capacity to the public cloud?  Think about it – if you can provide a fully cloud-compatible environment on great hardware that’s superbly managed, virtualized, and secured, why can’t you rent out any unused capacity you have at any given time?  Just like electricity co-generation, when I need more resources I draw from the cloud, but when I have extra resources I can sell it to someone else who has the need.

You might say that if your cloud environment is truly elastic, you’ll never have excess capacity.  Sorry, but things are never that easy.  Today large enterprises typically have very poor asset utilization, but for financial and other reasons dumping this capacity on eBay does not always make sense.  So, what about subletting your computing capacity to the cloud?

Then, if I take all of the big corporate data centers in the world and weave them into this open co-generation market, then instead of buying instances from Amazon, Citigroup can buy them from GE or Exxon.  What if you need a configuration that is common in the enterprise, but not in the cloud (e.g. true enterprise class analytic servers with 100TB capacity), perhaps you can rent one for a few days.  It may be more cost-effective than running the same job on 300 Ec2 instances over the same timeframe.

There may be many reasons why the co-generation cloud computing market may never evolve, but those reasons are not technical. Doing this is not rocket science.

With Amazon VPC now in the picture, the BI datasets are now as secure at Amazon as they are in your data center.  Why wouldn’t you use the cloud for your BI needs?

By integrating with the security, governance and compliance infrastructures of enterprise IT, VPC eliminates one of the primary barriers to cloud adoption for mainstream business computing. Sure, there are still going to be issues, but this was the big one.

I won’t rehash all of the offering details here.  You can read them on Werner Vogels blog and TechCrunch.

The hybrid cloud is a reality.  You can now integrate your internal fixed IT infrastructure with large external clouds with a high degree of integration with enterprise tools. VPC allows you to assign IP addresses, create subnets, and connect your existing data centers to Amazon using secure VPN technology. Sure, this is not the same level of connectivity via dedicated secure lines that most big outsourcers provide, but it’s pretty strong and many very smart people (including Chris Hoff at Cisco) are bullish from a security perspective.

I will think about this a bit more, but Werner Vogels makes the claim that “private clouds are not clouds” mainly because they are not truly elastic.  There may be some benefits to using Eucalyptus or VMware’s vSphere in your data center, but you still need to buy hardware and install it and that’s not cloud computing according to Vogels.

One thing that’s certain, the game has changed – again!  Amazon’s VPC is far and away the most significant cloud computing announcement so far this year, and I’m going to go out on a limb and predict that on December 31 it will still hold that distinction.

What do you think??

A stealth start-up called InfiBase has published some very interesting data on their blog recently. It makes me want to know more about them, so if you have the scoop let me know.

First, they have put out two posts on sites using Amazon EC2, with other cloud providers included in the last posting earlier this month. Here is their chart showing the top 500,000 sites by cloud providers.  Note how close Amazon EC2 and Rackspace CloudServers (based on Slicehost) are in this ranking.

Source: InfiBase

I was interested to see Joyent in third place, well ahead of both Google and GoGrid, and I wonder what this might look like a year from now.

In another post InfiBase performed a deep dive into the processing dynamic of various EC2 instances, including which processors are being used and how they stack up.  Here is just one of their great charts which shows that AMD processors are used at the low end of EC2 while Intel takes over at the very high end.

Source: InfiBase

With the data they are previewing in their blog (see the full posts there), I am intrigued.

However, when I see their current marketing and branding it makes me want to chuckle.  Instead of Salesforce, Successforce, and Force.com, they now market Sales Cloud, Service Cloud, and Custom Cloud.  They already had the cloud creds, but by trying so hard it makes them look a bit silly.  I wonder if this rebranding is hurting or helping their sales numbers…