So now what? I basically started by registering an account with Amazon Elastic Cloud Compute (EC2) and created a Windows environment in the cloud that would have taken me days if I were to provision them on physical servers back at my local data center. If I were a startup with little computing resources available, this option would definitely look quite appealing. I could setup my whole test environment within minutes and not even have to provision a single physical server. Initially, this all seems quite efficient but as I thought about the future from the customer’s perspective, what happens if their company decides one day to host their own applications in-house or at their local data center? Or what if budget is scarce and they no longer can afford Amazon EC2 pricing? Or what if one of their developers wants to test something on an EC2 instance but does not have access to the Internet? One of the main concerns with IaaS cloud platforms is virtual machine mobility. Once a customer deploys a virtual machine in the cloud, how do they get it out of the cloud?

Since Amazon EC2 virtual machines do not have a mobile, encapsulated virtual disk file similar to VMware’s Virtual Machine Disk Format (VMDK), customers simply cannot easily import and export virtual disk files. Technically, it is possible but it either requires purchasing additional third-party software or scripting complex Linux commands to process and convert EC2 files. If you are a Windows administrator, I’m sure you would want the easiest way possible.

To help in this situation, VMware just released a Knowledge Base article on how to convert EC2 Windows instances into VMware format using VMware vCenter Converter Standalone, which is available as a free download. The KB is available here: http://kb.vmware.com/kb/1018015. Do note that users should ensure that they have properly licensed their Windows virtual machines after exporting them from EC2.

This solution means users can now take a Windows instance from EC2 and run it locally on their desktop using VMware Workstation or one of VMware’s freely available hosted virtualization platforms, VMware Player or VMware Server. In addition, once converted, they have the option to migrate their virtual machines over to VMware vSphere or to simply upload the virtual disk to a VMware vCloud Express service provider. VMware vCloud Express is an IaaS offering delivered by VMware service provider partners which offers flexible, high-performance on-demand computing giving customers the power and control to configure resources exactly the way they want them.

Most users will find locked-down IaaS platforms like EC2 to be too restrictive, but users that initially started with EC2 because of its ease of creating virtual machines now have an escape route to more flexible VMware-based virtualization and cloud platforms.

A look at Citrix’s own Support Forums gives a clue about the popularity of Essentials for Hyper-V. Back in November,2009, this is what it looked like:

The numbers highlighted are the actual number of posts—a total of six posts. Maybe it was the holiday season. Well, let’s take a look at it today, almost four months later:

A 100 percent increase in postings! Ok, this post was meant to poke a little fun at Citrix, but in all seriousness, how many of you out there are actually using Citrix Essentials for Hyper-V? If you’re a Hyper-V administrator, don’t you already have tons of management applications to deal with?  Having another application to manage your back-end storage would just add to the complexity. If Essentials was so essential wouldn’t Microsoft be talking about how much customers need this product?

Stay tuned for an in-depth look into some of the technical reasons why administrators should think twice before implementing Citrix Essentials for Hyper-V.

vSphere currently supports 55 guest operating systems – maximum flexibility for deploying applications in a virtual environment

On the guest operating system support side, VMware ESX supports far more guest operating systems than any other bare-metal virtualization platform. In fact, VMware supports 55 different OSs ranging from Windows, OS/2, Solaris, and even NetWare (yes, a lot of people still use NetWare!) Visit the online VMware Compatibility Guide and you’ll see that ESX4 supports over 460 versions of those 55 OSs:

vSphere VMs scale up to 8 vCPUs and 255 GB of RAM – maximum scalability to take advantage of today’s hardware

What’s even more exciting is that for each one of those 460 versions of the 55 supported guest operating systems, ESX enables up to 8 virtual CPUs. Be it Windows or Linux, any ESX4 VM can be configured to have 8 vCPUs along with 255GB of RAM—the exception being Windows XP and Vista, which has an OS limitation of 2 vCPUs even when running natively on physical (Hyper-V on the other hand only supports up to 64GB of VM RAM). With this support, maximum throughput achievable from a single VM is much higher in ESX4 than in previous versions.

ESX4 even allows one to configure vCPUs in odd numbers such as 3, 5, or 7 if one chooses. This ability allows application workloads to be configured with the most flexibility:

Hyper-V limits choice, treats Linux VMs as second-class citizens and doesn’t scale to the full potential of today’s hardware

In contrast, Microsoft Hyper-V has a much smaller guest OS support list along with very limited vSMP support. Take a close look at their guest operating system support list and you’ll notice Windows 2008 and Windows 7 VMs are limited to only 1, 2, or 4 vCPUs. In addition, Hyper-V Linux VMs are hindered by only 1vCPU. Don’t even try running SAP on a Linux VM under Hyper-V because it’s NOT supported! SAP does not even plan to evaluate this type of combination for their applications. On the other hand, ESX has supported SAP in both Windows and Linux VMs since 2007.

In addition to Linux, legacy systems, even Windows legacy systems, are also treated as second-class citizens. Where’s the 4 vCPU support for Windows 2003? Or how about 2 vCPU support in Windows 2000? Can’t do it on Hyper-V!  Could Microsoft be limiting support for legacy Windows systems to “encourage” upgrades to newer OSs? Hmm – VMware does not have that type of conflict of interest. ESX4 does a better job at running Windows OSs than Microsoft by supporting practically every Windows version since 3.1 and even MS DOS 6.22! Alright, I know we don’t support Windows ME (Mistake Edition), but then again, who’s willing to admit that they’re running the worst OS of all time?

The Bottom Line

Virtualization should not be used as a vehicle to influence a customer’s choice of guest operating systems or when they should do an operating system upgrade. VMware strives to support guest operating systems as consistently and equally as possible. On the other hand, Microsoft’s preferential treatment of Windows Server 2008 and Windows 7 pressures companies to deploy these OSs in order to get the full benefits of Hyper-V. If a guest OS doesn’t generate any revenue for Microsoft, it most likely won’t show up on their guest support list. Do you think Solaris will ever make the list?

With the introduction of 6 core CPUs and with 8 cores just right around the corner, customers need a virtualization platform that can take advantage of the increased processing power. VMs under ESX4 does just that with support for up to 8 vCPUs and 255GB of RAM. On the other hand, Hyper-V comes up short and is not yet enterprise-ready, as noted pointedly by Burton Group.

Business Continuity is a Highly Valued Virtualization Function

Among the primary reasons why companies virtualize their applications is to improve business continuity – or in other words to make all applications as available as possible, as often as possible. Actually, we are now seeing a lot of customers value this virtualization capability even more than server consolidation. With vSphere, there are several established methods available to improve application availability; it all depends on the type of failure you want to protect against and the amount of downtime one can tolerate. For this post however, we’re going to examine VMware vSphere High Availability and a very helpful, vSphere exclusive, feature that enables an organization to prioritize which VMs restart first in the event of a hardware failure – pretty interesting stuff.

VMware HA Enables Unheard of Levels of Application Availability

As we all probably know by now, VMware HA has been a widely adopted solution to minimize application downtime in case of server failure. Once HA has been enabled on a vSphere cluster (directly from vSphere Client with just a few clicks), vSphere will continuously monitor the status of all VMs running in a specific cluster. Should a host failure occur, vSphere will then automatically restart the affected VMs on a different host, greatly reducing application Recovery Time Objective (RTO). With vSphere 4 we have extended the capabilities of VMware HA to enable automatic VM restart in the case of a guest operating system failure. So now, even if the hardware is fine, but a single virtual machine’s operating system fails, that application will restart. That is pretty good!

vSphere Exclusive Feature – Prioritize which Applications Restart First

That’s a cool advancement, but in terms of HA triggered by a server failure, we still felt that we needed to do better, because all applications on a host aren’t equal are they? The ability to prioritize mission critical apps to recover first, before tier-2 applications, could result in a lot of benefits to an organization. That ability could save a lot of money, could limit revenue loss, and could even protect a company’s reputation. Consequently, a primary objective of HA should be to further minimize the cost of downtime by prioritizing the restart of mission critical applications. Well, vSphere 4 has a solution for that – with vSphere, one can easily configure a failover restart prioritization level of a VM. It only takes a mouse click – or three.

By setting the failover restart priority level of a VM to “High”, vSphere will make sure the VM is restarted before those with a “Medium” or “Low” priority. Even better, the failover restart prioritization is a VM attribute, independent from the host, and will follow the VM even if you migrate it or restart it on a different host in the cluster.

This Feature Does Not Exist in Hyper-V R2

Even with Windows Server 2008 R2, it appears that Hyper-V won’t provide failover restart prioritization capabilities. Yes, with Microsoft Clustering Services, one could make a VM highly available – but when it comes to prioritizing VMs for restart, the user is really left with three insufficient alternatives:

1) Forget about automated VM restart and go back to manual restart – sounds fun…others have already written on this limitation

2) Spend time and effort creating and maintaining scripts for prioritized restart – this option sounds like a management nightmare considering that scripts would have to be updated every time a new VM is created or migrated to a different host

3) Run fewer apps per server so that you reduce the risk of extending the RTO of mission critical apps – but that increases the overall TCO of your virtualization infrastructure as you are then using more physical servers.

Or you could challenge Murphy’s Law and hope for the best. Yes, I am biased, but I think you’d agree that 1) The ability to prioritize VM restart in the event of failure is an important, valuable capability and 2) A single click in the vSphere console to prioritize VM restart beats any of these Hyper-V alternatives.

NIC teaming policies can be set for any of the supported networking cards (over 450 models) and allow users to configure multiple active and standby adapters. Teaming configurations can vary per port groups on the same virtual switch and uplinks.

Microsoft Hyper-V R2 will still not have integrated NIC teaming, instead relying on third-party NIC drivers to provide the functionality. The issues with the third-party approach are: 1) the drivers only work with NICs from that same third-party, 2) it requires a separate installation (and often other crazy stuff as vCritical showed in this blog post), and 3) it is unclear whether Microsoft or the third-party provides support should an issue arise.

Managing NIC teaming Hyper-V servers can quickly become a painful and time consuming activity. Besides having to install third-party drivers on each server, teaming will have to configured and managed locally on each server using the third-party management tool (like the HP Network Configuration Utility in the image below) that is not aware of your virtual switch configurations.

This is if you decide to enable Hyper-V on a full Windows Server 2008 installation, bringing the over 10GB of bloated Windows code base and all its security and patching related pains in your virtualization stack. Should you instead decide to follow Microsoft’s recommendation and enable the Hyper-V role on Windows Server 2008 Server Core deployment or use Hyper-V Server (i.e. Windows Server 2008 Server Core with Hyper-V enabled) in an attempt to reduce that code base to “just” 3-4GB, well, make sure to cancel your dinner plans and allocate few more hours to setup teaming via command line!… but, hey, that’s the “Windows you know”.

Built-in NIC teaming isn’t the only aspect that vSphere does better than Hyper-V when it comes to networking management. Directly from the vSphere Client users can also set up policies for networking load balancing, layer-2 security and networking traffic shaping.

When Microsoft was rolling out Hyper-V and Quick Migration, they were adamant in telling users that Quick Migration was all they needed.  I clearly remember their representatives claiming that migrating a VM should be a rare event, so the downtime was tolerable.  Hyper-V users were told that if they needed to migrate VMs more often than once a month, they were at fault for making mistakes in their capacity planning and VM placements.  It was a very transparent attempt to downplay the absence of true live migration in their product.  Microsoft was effectively telling customers, “You don’t really need that live migration stuff.  It’s overkill and nobody really uses VMotion anyway.”

You can’t blame Microsoft for making excuses for Quick Migration, but claiming that nobody uses VMotion just elicited chuckles from the audiences at their conferences — most whom indicated they were VMware users whenever a “show of hands” poll was taken.  The truth is 70% of VMware customers are using VMotion in production and they rave about the flexibility it gives them.

The truth comes out

Now that Microsoft has live migration on their Hyper-V roadmap, we’re starting to learn that they never really thought much of Quick Migration themselves.  I heard Mark Russinovich, one of their technical luminaries who gave a talk about Hyper-V R2 futures, actually come out and say what we all knew to be true, “Quick Migration was our first attempt to do a live migration, and to put a nice spin on it, we called it Quick Migration. [...] Even though we said, [...] ‘trust us this is really cool, this is what you want, you don’t want instant, that’s not as good as this, this is quick,  but people didn’t seem to buy that, so we ended up [...] implementing live migration, so that Quick Migration stuff is crap, this is really good.’”  Mark’s honesty got some laughs from the packed session, but it puts us on notice that we should be skeptical when Microsoft tags a feature as “Quick”.

Microsoft’s new “Quick Storage Migration” promised for SCVMM R2 is just such a feature to approach carefully.  Microsoft is positioning it as equivalent to the VMware vSphere Storage VMotion feature, which does true live migration of a VM’s virtual disks to another storage location with zero VM downtime.  As you might infer from the name, “Quick Storage Migration” forces a Hyper-V VM offline during the move.  The downtime occurs while a snapshot made of the VM at the start of the move is reverted back to the primary virtual disk once it’s copied to the new storage location.  Microsoft didn’t say how long that downtime would be, but as we saw with Quick Migration, anything longer than a few pings will result in timed out connections.

Look out, more “Quick” is on its way

It’s not surprising to see Microsoft reaching for their “Quick Migration” marketing playbook as they try to explain away the limitations of Quick Storage Migration.  Once again, they’re asking customers to accept “Quick” as good enough and they’re even laying a guilt trip on anyone who has the temerity to ask for true live storage migration.  In Jeff Woolsey’s session at TechEd, we were told not to worry about the downtime because, “How often do you do maintenance on a SAN?  Hopefully that answer is very rarely, because if it’s any more than that, then you probably misconfigured your SAN.”  VMware customers who have been using Storage VMotion since VI3.5 would certainly disagree.  They use Storage VMotion for lots of reasons including rebalancing storage, relocating VMs when business units reorganize, and maybe they do misconfigure a SAN once in a while, but with Storage VMotion, fixing it is no big deal.  With Microsoft’s Quick Storage Migration, you’ll need to schedule a maintenance downtime window for any of those use cases.

By now you’ve probably detected a pattern in Microsoft’s Hyper-V/SCVMM maturation as shown by these “Quick” features.  First, they roll out a deeply flawed “me too” feature while telling customers their implementation is good enough and they don’t really need the fully baked version available from VMware.  Next, any customers asking for the full feature are told they only want it because they are doing something wrong in managing their systems.  Finally, when Microsoft gets around to offering the real deal, customers are told that the initial, crippled feature was “crap” all along.

Don’t let them “Quick” on you again

Fortunately, virtualization users don’t have to put up with such treatment.  VMware has been delivering virtualization innovations, rather than asking customers to meekly wait while they catch up to the other guys.  Once Microsoft has a look at the breakthroughs in vSphere like Fault Tolerance, vShield Zones, Data Recovery, Distributed Power Management, Host Profiles and vNetwork Distributed Switches, I wonder which of them will get the “Quick” treatment in some future Hyper-V release.

Oracle VM lacks a number of important features and tools that are required to run applications in the enterprise.  Absence of these features in any virtualization solution makes for an enterprise environment that may not be able to meet SLA (Service Level Agreement) obligations to the customers as well as an administrative headache to deploy and maintain.

Here are some of the Oracle VM shortcomings we’ve encountered after trying it out:

• Oracle VM has no built-in capability to integrate with naming services such as Microsoft Active Directory or LDAP.  We all know the shortcut that leads to – just have everyone share the same login.
• Oracle VM has limited RBAC (Role-Based Access Control) capability so you can’t control user permissions with fine-grained customizable access to various objects in OVM.
• Limited guest OS support (OEL, Red Hat, Windows) rules out Oracle VM if your applications run on other OSs like Novell NetWare, SUSE Linux or Solaris.
• Oracle VM has no real workload balancing tool like VMware DRS.  You’ll need to run your servers at lower utilization and keep a closer eye on VMs with spiking loads.
• Oracle VM’s reliance on Xen as its virtualization engine means lower VM density per host as Xen 3.1 has no memory overcommit feature like that of VMware ESX.  Overcommit lowers TCO and add great flexibility to VM management.
• Oracle VM pools are only server pools and not resource pools. This prevents memory and CPU resources from being better managed and shared by VMs in the pool.  You can’t carve out resource pools for your business and let them operate autonomously as you can with VMware.
• Oracle VM has no built-in fault tolerance features and VMs must rely on third party tools for zero-downtime fault tolerance.  The new VMware vSphere FT feature brings continuous availability to any OS, any app, any hardware.
• Oracle VM has no snapshot features. That means developers and administrators are handicapped as they are unable to make instant point-in-time copies of VMs to use to roll back development and administrative activities.
• No patch management utility in Oracle VM means manual intervention or extensive scripting is required to administer patching of hosts and VMs.  VMware Update Manager automates host and guest patching – typically the most time-consuming task for virtualization sysadmins.
• Oracle VM also has no backup utility for VMs and relies on LAN-based backup agents that run in each VM. It is also limited in its capability to take advantage of Storage Array based backup for VMs. Without a snapshot-based backup proxy feature like VMware Consolidated Backup or the new vSphere Data Recovery, backups that run in the guests tie up your network and burn host CPU cycles.
• NIC teaming or bonding support is also missing in Oracle VM and it relies on NIC vendors for support or extensive scripting to provide network redundancy.  NIC teaming and bonding is built into VMware ESX and dead simple to use.
• Performance monitoring in Oracle VM is limited in the statistics gathered and the time frames that can be viewed.  That makes it insufficient for capacity planning or long-term resource utilization monitoring.

Finally, let me say a word or two about Oracle VM Templates.

Virtual appliances like Oracle VM templates are a good way to distribute software in standardized and portable container.  The virtual appliance concept is even better when customers can choose the applications, operating systems and hypervisors to use with those appliances.  However, Oracle has taken the virtual appliance concept and twisted it into another way to lock its customers into a 100% Oracle world.  The point of virtual appliances is simplicity and flexibility, but Oracle templates take away the flexibility benefit entirely.  No prudent DBA, system administrator or IT Manager that I know of would deploy these locked-down templates as gold image in their production enterprise datacenter.  Those users need appliances that can be adapted to their own requirements Those customizations can be choice of OS, file system layout, performance tuning kernel parameters or security requirements for package deployments. The restricted Oracle VM templates at best can only serve for preliminary validation of Oracle’s products.  Their templates have a fixed configuration and they only support Oracle’s own OEL operating system and they can run only on the Oracle VM hypervisor platform. It sounds like the perfect strategy for a top-to-bottom lock-in.  Not many CIOs will want to accept that degree of control by a single vendor.

It will be interesting to see what impact the purchase of Sun Microsystems will have on Oracle VM’s roadmap. Sun brings their own Xen-based hypervisor to the party with xVM and it’s supposedly a pretty good implementation, however, I have yet to see a copy of Sun xVM in the wild.  Will Oracle keep both Xen products alive or is one headed for the dustbin?

We already knew that XenServer5  had a shortcoming that is showstopper for most enterprises – every user with login access to the management UI gets full root access to all management hosts and VMs.  When prospective customers ask us how VMware Infrastructure compares to our competition and we tell them about this one issue, we see their eyes bug out in surprise, especially if they operate in a security and audit conscious environment like a bank or government agency.  They simply can’t consider a virtualization platform without access controls and audit tracking of logins and configuration changes – features that VMware vCenter has provided for years.  Burton Group’s Chris Wolf made this issue his primary reason for rating XenServer “not enterprise production-ready”.  In fact, only VMware ESX made the production-ready cut in Chris’ ratings.

Would the now free XenServer, managed by the not-free Citrix Essentials for XenServer, patch up that gaping security hole?  The Citrix Essentials for XenServer trial download just recently became available and, once installed, I quickly saw they had not fixed the issue and XenServer will stay a  liability in any enterprise datacenter.

XenServer root-access only is a critical security flaw

After I connected to my XenServer 5 Server Pool using XenCenter (strangely enough, despite all the fuss about the Citrix Essentials management tools, the XenServer management console is still called “XenCenter”), I was amazed to see I was still allowed root console access to the hypervisor. It seems that there is no way to create accounts other than root that can use XenCenter to connect and manage the virtualization environment. Of course I don’t need to explain what a critical security flaw that can become if the XenCenter console is compromised – the attacker gets the keys to the kingdom – hosts, VMs, everything. Also , it seems that there is no built-in feature to integrate XenServer with any naming services such as Active Directory or LDAP and you may have to buy additional licenses for third- party software (Citrix mentions Centrify in their documentation) to provide that service for your XenServer  environment.  It appears that even though an extra cost third-party directory service connector might let you control who can access XenCenter, every user granted that privilege still has full root access to the entire XenServer environment.  We’d like to hear from anyone who’s tried XenServer with Centrify who can verify this all-or-nothing situation.

XenServer’s lack of RBAC is a critical security and operational shortcoming

RBAC ( role-based access control) is an approach to restricting VM, hypervisor or pool access to authorized users. Within a virtualization solution, roles are created for various job functions. The permissions to perform certain operations are assigned to specific roles and those roles are then granted to users registered in your directory service. Some useful  pre-defined roles you can choose in vCenter are Administrator, Manager, Virtual Machine User, NOC operator  or Read-Only, and you can also created fine-grained custom roles..

XenServer has no such capability and  gives all users  the same root-level of control over all objects. This can create an administrative nightmare as you are unable to delegate limited privileges and assign roles to  various members of your organization where granting full privileges  is not allowed.

RBAC is now integrated by many OS and application vendors in their products to support financial, government and businesses customers who have made it a mandatory feature for  managing their large networks.  Those users don’t allow  components lacking RBAC into their environments.  Unfortunately for Citrix, XenServer seems to be one of those products that will remain off-limits.

As we continue to look at XenServer and Citrix Essentials, I’ll point out in upcoming posts a number of shortcomings seen in our initial hands-on evaluation that will demonstrate why the combination falls short of enterprise datacenter requirements

I think it was always theoretically possible, like time travel or the yeti, but I don’t think there has even been documented proof before – until now.  As Stu has written on vinternals – and there is no sense in me trying to recreate the details here, he’s written an excellent article – a Microsoft technical case study, internally written and available on TechNet, summarizes the experience a Microsoft IT department had when deploying Hyper-V.  It is surprising that the Microsoft paper ever saw the light of day, as it essentially lays out a strong case for why, contrary to Microsoft’s ubiquitous marketing claims, Hyper-V 1) is not equivalent to a VMware solution and 2) is not ready for use in a production datacenter – at any cost – for any workload.  Is Hyper-V the new Corvair? The Microsoft case study actually validates a number of the lab findings we have made available on Why Choose VMware and “un-validates” much of Microsoft’s own virtualization marketing and myth-busting rhetoric.

The external validation is nice, we just never thought it would come from Microsoft

I do have to say that it is nice to have some external validation of what we’ve found to be true in our lab analyses and of what many of our customers have told us regarding Hyper-V.  At times we feel like we may be buried under a tidal wave of Microsoft marketing spend. (They do have a cool robot though!) We just never thought that Microsoft itself would be our ally in refuting Microsoft’s marketing rhetoric and in getting to a more accurate, fact-based comparison of currently available products.  Thanks guys!  Myth-busters-busted.

The low consolidation ratios make Hyper-V even more expensive than we had thought

And one more thing, as this paper relates to our Cost Per Application calculator, and our findings that due to ESX’s ability to drive a higher consolidation of workloads on a single host, VI3 is actually less expensive than Microsoft’s Hyper-V offering – with only 1-2 additional virtual machines per host.  Microsoft’s paper states that when using Hyper-V in production, on a server with 8-12 processors and 32 GB of RAM, they were able to run only 5-6 virtual machines.  In our calculator, we had assumed Microsoft Hyper-V could support a pretty standard 1-1.5 machines per core, or 12 virtual machines on that type of server.  Well, we gave Hyper-V too much credit.  With such limited consolidation on Hyper-V hosts, Hyper-V is way more expensive than even our top of the line VMware VI3 Enterprise edition than we had originally calculated.  And VI3 is a proven solution with a lot more functionality.   Perhaps we should rerun the numbers? Perhaps it doesn’t matter?

One last thing

Sorry, one additional item, before all you conspiracy theorists out there start speculating.  We did not infiltrate Microsoft’s IT department and brainwash or bribe someone to write and publish this paper…so I hope we can put an end to that theory right now. Although I did hear that Oliver Stone is now working on a screenplay.

http://www.ntpro.nl/blog/archives/1043-When-Im-feeling-blue.html

Would you run your mission critical, production applications on a platform this instable? I wonder what would happen if a normal user with VM creation rights provisioned a VM and accidently stored it on a CIFS share. Hmmmm…..Microsoft may need to rename their mascot IT 22-7.