In the interview, John interviews:

• Rich Miller, Managing Director & Principal, Cumulati
• Bernard Golden, CEO, Hyperstratus
• Randy Bias, CEO, Cloudscaling

The link is here: http://www.justin.tv/nicefishfilms/b/269320590

The video clip is below:

Watch live video from #theCube from SiliconANGLE.tv on Justin.tv

Randy’s comments are mostly in the 9:00 to 35:00 minute window but the audio was off from 9:00 – 17:22 so it’s a bit faint in the first half.

Some of the questions that are addressed in this talk include:

• What is the current landscape for cloud computing?
• What are the issues now?
• Which issues are key?
• Is security a real issue?
• How do I use cloud while maintaining my compliance?
• Who is using cloud inside the enterprise? Apps teams or Infrastructure teams?
• How should cloud service providers think about cloud?
• How big of a deal is cloud?
• What are the entrepreneurial opportunities?

Thanks to John Firrier and the SiliconAngle.tv team for a great interview.

Relax after VMWorld and come share Dos Equis with the OpenStackers.

We’ll be discussing cloud, cloud, cloud and maybe dodgeball Thursday evening.

5PM-8PM Sept. 2nd
@ Tres Agaves Restaurant
130 Townsend Street
San Francisco, CA 94107
(Just a few blocks from Moscone)

Come out and meet people working on OpenStack. (We’re interesting people and willing to talk about things besides cloud. Promise…)

View Larger Map

Stay Open My Friends

Randy Bias discusses the impact of OpenStack on the cloud computing ecosystem with sandhill.com’s Kamesh Pemmaraju.

Kamesh outlines the current state of cloud adoption, OpenStack and looks ahead.

Once you have swift services running, you are going to want to interact with them. (You can have them running in minutes if you want: swift-solo)

Most of the currently available tools that interact with Rackspace’s Cloudfiles are hardcoded to that API endpoint, so even though the APIs are virtually identical, most the current releases don’t work with swift yet. Since we needed something for testing and demonstrations, we’ve been patching some of the third party projects to store and retrieve files with swift. Here’s a couple quick examples using Cyberduck, a GUI program, and Fog, a Ruby gem, from the command line.

Setting up Swift

All of the tools expect interaction with SSL enabled servers, so we will need to install and enable that within swift.  In the swift-solo repo, edit chef/cookbooks/swift/attributes/swift.rb and enable ssl:

default[:swift][:proxy_server][:use_ssl] = true
default[:swift][:auth_server][:use_ssl] = true

default[:swift][:repository][:url] = "http://github.com/cloudscaling/swift.git"
default[:swift][:repository][:tag] = "cyberduck"

ubuntu@host:~/swift-solo$ sudo swift-auth-create-account account user password
https://example.com:8080/v1/338b6b2d-5137-40b4-9b95-9106a0d4db52

ubuntu@host: st -A https://127.0.0.1:11000/v1.0 -U account:username -K password stat
Account: 338b6b2d-5137-40b4-9b95-9106a0d4db52
Containers: 0
Objects: 0

Using Cyberduck

Using fog

:default:
:rackspace_api_key:     password
:rackspace_username:    account:user
:rackspace_auth_url:    example.com:11000

username@host$ fog
Welcome to fog interactive!
>> f = Fog::Rackspace::Files.new(Fog.credentials)
>> f.put_container("test_container")
>> f.get_container("test_container")
>> f.directories
<Fog::Rackspace::Files::Directories
[
<Fog::Rackspace::Files::Directory
key="testcontainer",
bytes=10,
count=1
>
]
>

UPDATE:
The lastest builds of Cyberduck now support OpenStack

OpenStack is a cloud computing framework for building infrastructure as a service, and Swift is a subproject which provides a scalable distributed object store. We have already expressed opinions about how a community of service providers all running an open platform would do to the cloud computing ecosystem (think Linux kernel of cloud). To support our own work, we try to make it as simple and easy as possible to build and configure clusters. We want to make it simple and easy for everyone else too. Today we are releasing ‘swift-solo‘ on github, which is essentially a chef automated install of the instructions for setting up an Ubuntu 10 VM for swift development. (We took inspiration from Opscode’s OpenStack Swift cookbook, but you’ll see a lot of things have been split out and extended. Thanks to Adam Jacob for those cookbooks and chef.)

The way it works now, after installing git and chef, one command will deploy and start swift on a single machine. The chef recipe can be configured to sync code from a git or bzr repository so you can quickly deploy changes in your development environment. There are also scripts to setup testing (basically install python-cloudfiles) and make sure the service is working (store an object and retrieve it).

Swift is actually 5 separate services, object, container, account, auth and proxy. In theory, each of those can be scaled separately, but in practice the documentation says Rackspace runs the object, container and account services together and breaks out the proxy and auth.

Object, container and account servers are members of partitioned consistent hash rings mapping names to physical locations. Object servers index the actual blobs, container servers index the objects and the account servers index the containers.

Auth servers provide authentication and the proxy servers are the API gateway that coordinates request and responses with the rest of the services.

There are some other moving parts to do things like handle replication, auditing the integrity of objects and rebalance the rings.

The Swift documentation has a lot more information about the architecture, operational considerations and the code. Dig in.

Cloudscaling is interested in creating an open cloud ecosystem and we’ll keep sharing our work and trying to lower the barrier to participation. In the future, we plan to explain and open some of the work we are doing to handle authentication, automating the installation/configuration across clusters and other operational lessons we learn along the way.

Your first running Swift service is only a git clone away.

We would love to hear your feedback, suggestions and issues.

Finally, a shout out and thanks to the Scalers working on OpenStack/Swift: Big ups to Caleb Tennis for working on these chef recipes, Blake Barnett, and Joe Arnold.

Want to hack on OpenStack or Chef? We’re looking for good people to build open source automation tools and cloud infrastructure. Join us.

Drop us a line…

The discussion will be moderated by Dana Gardner, principal analyst at Interarbor Solutions, and prolific ZDNet blogger and is being hosted by our friends over at rPath.

Join us this upcoming Thursday, August 19 at 11:00am PT / 2:00 PM ET for “State of the Cloud II: Enterprise Patterns Emerge”

We also invite you to submit questions before and during the event via twitter using the hashtag #SOTCII.

Register now to reserve your spot for what will surely be a vibrant debate!

Let’s dive in and I’ll explain.

Cloud Stack Evolution & ‘Camps’
Amazon Web Services (AWS) spawned a huge ecosystem of knock-offs, management systems, tools, and vendors. They include, but aren’t limited to:

• AWS API compatible ‘cloud stacks’ including Eucalyptus, Open Nebula, and others
• Cloud management systems for the AWS APIs and services such as RightScale and enStratus
• Cloud services layered on top of AWS services such as Jungle Disk (S3), Heroku (S3, EBS, EC2), and more

Prior, I wouldn’t have called the AWS ecosystem a ‘camp’ per se, but if you read our most recent article on Google’s foray into cloud storage, you know that it seems likely they will provide a 100% compatible version of S3 and EC2 this year. Imagine the impact of Google Compute & Storage with Amazon Web Services compatible APIs. Already the Google Storage API is nearly 100% compatible with S3.

Together, as a block, Amazon and Google could create a de facto duopoly for infrastructure clouds, which isn’t good for anyone. We need competition and more than two major players.

Up against the Amazon camp is VMware. In my article on Amazon vs. VMware last year I highlighted how these two businesses were on a collision course. Nothing has changed and competition is mounting between them. The reason is that telcos and service providers are under increasing threat from Amazon and soon Google. They need viable solutions and VMware is attempting to provide a competitive ecosystem.

The VMware cloud initiative, vCloud, is designed to arm enterprises and service providers to be competitive, but has not quite delivered yet. VMware has had a number of problems providing a full cloud stack. The software, now in beta, is codenamed ‘Redwood’ has had significant delays in getting to market. Their strategy for cloud infrastructure does not appear unified outside of delivering compute virtualization.

VMware, as a business, understands they need to make their customers competitive. They have made a number of strategic open source acquisitions such as SpringSource, RabbitMQ, and Redis. There are also murmurings that they have some special projects inside that are ‘up the stack’ from their virtualization offerings. In total this shows that VMware ‘gets it’ in that they want to create a competitive ecosystem. While each of these is currently a point solution, there is yet to be a coherent story here. Can VMW build a consistent story and strategy around these disparate pieces? Only time will tell…

Besides these two camps, there is a long tail of clouds running various frameworks vying to establish themselves such as Cloud.com’s CloudStack, 3Tera, Hexagrid, Abiquo, OpenNebula, etc. John Treadway recently had posted a roundup describing all of the various cloud stacks out there.

OpenStack is stepping into the ring as a viable third camp. In particular, the OpenStack Storage solution is a clear contender to Amazon S3 & Google Storage. Many service providers and telcos have struggled to find a viable solution using commodity hardware that was price competitive. Suddenly, there is a viable proven solution.

Yet this is only storage. How can it create an effective ‘third camp’ alternative to Amazon and VMware for an entire cloud?

Lock-in, Architecture, Standards and The Truth about Interoperability

Interoperability for infrastructure clouds is poorly understood. Most believe that the problem lies in the on-disk image format (e.g. VMDK vs. VHD vs. qcow) or in the ‘hypervisor’ (although people don’t really unders/tand what this means). The truth is that lock-in has little or nothing to do with disk formats or the hypervisor. Most on-disk image formats are simply representations of block storage (i.e. disk drives). That means converting between a VMware VMDK and a Citrix XenServer/Hyper-V VHD is relatively trivial.

What about booting the converted disk image up on a new hypervisor? Guess what, since most hypervisors now rely on hardware virtualization (HVM) [1] using Intel-VT/AMD-V, that means that by default most will work with unmodified operating systems out of the box. No changes needed. The only downside of this is that usually the resulting performance is poor. This requires new paravirtualization (PV) drivers in the converted image. What does that mean? After converting the image from one format to another, you simply have to install the PV drivers for the correct OS. A process that requires being methodical, but is in no way technically challenging.

Where is the lock-in then? If it’s not the hypervisor, what makes moving from one cloud to another so difficult? Simply put, it’s architectural differences. Every cloud chooses to do storage and networking differently.

For example, if you wanted to move a virtual machine from GoGrid to Amazon, converting the GoGrid image to an AMI is not difficult. Unfortunately, GoGrid uses two networks, a ‘frontend’ and a ‘backend’ where your cloud storage system is connected to via the backend network. Every Amazon virtual server has only a single network interface. If your application assumes a separate backend network then what happens when it moves to a cloud without one? Or vice versa? Similar architectural incompatibilities exist between Rackspace Cloud, Savvis, Terremark, Hosting.com, Joyent, and all of the others.

The problem here, to be a bit more succinct, is that we need reference architectures for how infrastructure clouds are built. Amazon is one such reference. VMware’s vCloud is potentially another. Now there could be a truly open option with the gravity to gather community support.

More on The Third Camp

OpenStack’s potential to build a real community and a set of reference architectures drives towards greater standardization and interoperability. Perhaps more important than a cloud storage alternative, is this possibility for a true OpenStack community to form a critical mass such that a similar level of developers contributing to it as Amazon or VMware. Then commercial and alternative offerings, such as Cloud.com, Hexagrid, and OpenNebula can match their APIs and architectures to this set of reference architectures.

Will it happen? It’s hard to say, but the opportunity is there. Rackspace and others are putting serious weight behind this initiative.

What This Means for Telcos and Service Providers

For Telcos and SPs this means an alternative to VMware’s vCloud for commodity service offerings. A way to compete and operate at scale like Amazon and at a similar price point. Standardization through a similar reference architecture means greater compatibility between service provider clouds, which means greater benefit for customers and less lock-in, making them more desirable than the walled gardens.

You don’t want to differentiate on the basic compute, storage, and network offering. You want this to be as standard and interoperable as possible, just like 3G networks, TCP/IP, and similar service provider technologies. By creating a common open platform that everyone uses there is a better opportunity to facilitate wider adoption, create a competitive infrastructure service marketplace where providers work on differentiating in areas where they have an inherent advantage:

• Service and support
• Network & datacenter tie-ins (e.g. MPLS, hosting/co-lo)
• Bundled service offerings
• Differentiated value-added cloud services (VACS)

This is a game that all telcos and service providers understand. They have been playing it for the past 15+ years.

Conclusion

OpenStack, with a strong community behind it, should be an important tool for service providers and large telcos to compete at scale with the Amazon and Googles of this world.

We believe OpenStack and the reference architecture(s) associated with it will allow service providers (SP) to get their undifferentiated cloud offerings up and running early. For this reason, Cloudscaling will put real resources into supporting this effort. Getting basic cloud offerings up early then means providers can focus on support, services, bundling, and differentiated services as soon as possible, while embracing as large a customer base as possible. This is just as they compete on top of basic TCP/IP services today.

[1] Clearly, the market leader, Amazon, does not use HVM. They use PVM, a fully paravirtualized mode of Xen. However, even they seem to understand that HVM is the future. Their latest offering, designed for HPC, which is performance sensitive, uses HVM and supports unmodified operating systems. The reality is that the Intel-VT and AMD-V capabilities on the latest round of processors is incredibly fast and will only get faster. The battle is over. HVM and silicon won in this case.

• Amazon’s EC2 is clearly generating Real Revenue (TM) and could be at 500-750M in revenue for 2010
• Google has a massive global footprint and is north of one million servers
• The support structure for these servers includes a huge investment in datacenters, networking, and related
• The Googleplex houses an extremely large number of talented engineers in relevant areas: networking, storage, Linux kernel, server automation, etc.
• Google Storage recently went into BETA and is accepting developer signups

This last is perhaps one of the more telling signs. As you may be aware, Amazon’s Simple Storage Service (S3) pre-dates Amazon’s Elastic Compute Cloud (EC2). When Amazon launched in Europe they first deployed S3 followed by EC2. The same happened with their Asia/Pac deployment.

Amazon has built AWS in such a way that all of the services are synergistic, but in particular, EC2 is dependent on S3 as a persistent storage system of record. EC2 AMIs originate from and are stored in S3, it’s the long term backing store for Elastic Block Storage (EBS) and EBS snapshots, and it’s safe to assume that many other kinds of critical data that AWS relies on are stored there.

Would Google take a different approach? It’s doubtful. Amazon’s S3 is built to be a highly scalable storage platform[2]. Google’s own GoogleFS and BigTable server similar purposes. It’s certain that Google would use related design principles and hence we could see the Google Storage as a prelude to a Google on-demand virtual server service (Google Servers???).

Combined with the rumor I heard from a reasonably informed source I think we can look forward to an EC2 competitor, hopefully this year.

What I want to bring to folks attention here is that if another credible heavyweight enters into this market it will have a tremendous impact in further driving the utilitization of cloud services. In the medium term it will also threaten hosting providers and ‘enterprise clouds‘.

Why? I think what many hosting providers fail to understand is that Amazon and Google, particularly if fueled by direct competition, must grow up into the enterprise space. Just as in the Innovator’s Dilemma, they will eventually provide most of the features of any ‘enterprise’ cloud, which means that if you aren’t building to be competitive with Amazon and Google, you aren’t in the public cloud game.

Much more detail on this in a future posting.

Chuck’s blog posting covers the discussion fairly well and he was very helpful.  My final takeaway is that I think there can be a place for EMC, traditionally a ‘premium’ vendor, in even low cost commodity clouds.  The challenge however, as he rightfully identifies is that EMC, the business, has a hard time understanding these requirements.  The local EMC sales team I’ve been dealing with doesn’t really understand the inherent assumptions I’m making.  There is a lot of push to simply purchase the ‘bigger box’.

A strongly recommended read for folks trying to understand both EMC’s potential value proposition in the cloud and how to build ’scale out’ commodity clouds.

It’s hard for folks to internalize how things are changing in Internet-land, but I think you’ll get closer through this presentation.  It’s not the same-old, same-old any more. Cloud computing is the biggest change to how IT functions since the 1980s and the advent of the personal computer (and hence the rise of client-server/enterprise computing).

Enjoy … (and outstanding job, Andrew!)