Planning & architecting

This is the most importance phase and one you can’t afford to screw up, I chose to virtualize Domino so it was necessary to make sure I had a virtual environment that could handle the taxing resource demands that Domino servers require. Our environment is about 1,500 total users and somewhere between small and medium, we have many access methods including LAN, WAN, internet and both Notes Client, web-based, Blackberry and other smartphone users. I planned for 2 clustered servers at our corporate HQ, 1 server in the DMZ for Notes Client passthru, Lotus Traveler and SMTP, 1 Blackberry Enterprise Server and 1 server at our DR warm site. Web access to email from the public internet was a requirement so the challenge was trying to find a secure solution for that. I focused on 2 solutions for that, using a hardware reverse proxy device in the DMZ and using a server running reverse proxy software. I looked at hardware solutions from F5 Networks who make very nice hardware that is very customizable and integrates well with Domino, unfortunately we didn’t have the budget for it though. So the solution I chose was using IBM Websphere Secure Proxy Server running on a Windows server in the DMZ.

Next it was time to find hardware to use, I needed one new virtual host at corporate HQ, one at the DR site and more storage. We have a Hitachi AMS-500 SAN (2GB fiber) that we could add storage to as an option, but this was expensive and didn’t get us much storage for our budget. Instead I looked at lower cost iSCSI solutions, in particular the MSA line from HP, they are very affordable, easy to use and perform very well. My biggest concern though was making sure it had enough performance for the very intensive disk I/O demands that Domino had. So I did some research looking at IOPs numbers for the MSA and comparing them to Iometer tests that I ran inside a VM running on the AMS-500. I found that the MSA had equal or better performance compared to the AMS-500 so I chose that instead. We were able to obtain an iSCSI MSA with dual controllers, an extra shelf loaded with 450GB drives for a total of about 10TB raw storage and were able to stay in our budget. For servers we went with HP DL-385 G6 servers with AMD 6-core processors.

Now with Domino there are a couple of important hardware considerations that you should be aware of. First is Domino loves memory, it will use everything you give it as it does a lot of caching. So having an adequate amount of memory is a must. The next is with CPU’s, Domino  takes full advantage of the new AMD RVI and Intel EPT  technology in newer processors so having this is a must. In fact this can make a huge difference in performance compared to processors without it. Finally when it comes to virtualization using vSphere was a must, there have been numerous performance improvements in vSphere that are very beneficial to Domino. IBM has published some tech notes in the past about high CPU usage on virtualized Domino servers compared to physical ones with identical workloads. They point out that you should make sure you have suitable hardware and the proper architecture but also seem to place the blame on the overhead of the virtualization layer. They go as far as to not recommend virtualizing larger Domino workloads. Well the combination of newer CPU technologies and vSphere now make that possible and allow almost any Domino workload to be successfully virtualized. So to summarize:

• Only use CPUs with AMD RVI & Intel EPT
• Have plenty of memory, don’t plan on using overcommitment
• Make sure your storage can keep up with Domino
• Make sure you are using vSphere and not VI3

Next up is building the hardware including the MSA and vSphere servers and configuring the iSCSI storage and networking. Some additional bonuses with using vSphere is the many improvements to iSCSI including support for Jumbo Frames, faster initiators and support for multi-pathing. So stay tuned for part 2…

• DNS is the most critical component in your environment, for almost anything to work properly you need a DNS server up first. If you have all your DNS servers virtualized and they are on shared storage its going to be very difficult to bring them up because everything else that needs to come up first usually relies on DNS. Therefore you should make sure you have at least one DNS server on the local storage of one of your hosts, that way you can get it up early on and not have to wait for your shared storage to come up. If you want to go a step further keep one as a physical server also, so you can quickly and easily get DNS up right away.
• Active Directory, DHCP & any other authentication servers are also very important, having a workstation up and running is very handy so you can centrally connect to hosts and power things back on. If you’re using DHCP for your workstations and servers you again will want one up as soon as possible so you can get them on the network and get going. Not having an Active Directory server available can make Windows servers take a very long time to boot. So again, keep an AD & DHCP server on local storage or on a physical server so you can quickly get them up as soon as possible. AD, DNS & DHCP are critical to a Windows environment and without them available you’ll find that the rest of your environment is mostly useless.
• Know your ESX command line, if your vCenter Server and other workstations are not available you’ll need to start VM’s using the command line. Even if your DNS server is on a local VM you won’t be able to start it without the vSphere Client. Therefore you’ll have to log into the ESX console and manually start it, if you don’t know the command to do this that could be a problem. Keep a cheat sheet by your hosts with the basic commands that you’ll need like vmware-cmd to get things up and running using the console.
• Know your host IP addresses, if DNS is not up yet you won’t be able to connect to your hosts using putty or the vSphere client using their host names, you probably won’t know their IP address and without a DNS server can’t look them up. Therefore keep a list of your host IP addresses so you can use that to connect to them.
• Know how to re-scan storage, your hosts may come up before your shared storage, once your shared storage is up you’ll need to rescan from your hosts so they will see it and you can restart the VM’s on it. You can do this using the vSphere client by clicking on Configuration, Storage Adapters, selecting your HBA and clicking the Rescan button and then select search for new devices. You can also do this using the command line esxcfg-rescan utility.
• Make sure you know where your datacenter keys are if you use a electronic card scanner to open your doors. Most systems are placed in the datacenter and if the power goes out your doors are not going to work. There is nothing worse than running around trying to find keys in a crisis to get into the datacenter. And make sure you don’t keep the keys in datacenter or you’ll have to break down the door to get in. (thanks Tony DiMaggio for reminding me about this one)

Being prepared is critical in crisis situation to ensure you can react quickly to get things back up and running. Sometimes it takes a crisis to point out any shortcomings that you may have in your environment but thinking ahead can save you from big headaches later on.

So go check out the new version of the vLaunchpad and let me know what you think, just remember it’s not done yet and I have lots of changes to make to it. Thanks to the Netherlands gang (Duncan, Eric & Gabe) for their early feedback that helped me get the design right.

• mcgivern_jon at emc dot com from Westborough, MA
• brian_alexander at mentor dot com from Longmont, CO

David Davis from TrainSignal will be contacting you, thank you for your participation in the voting!

Position	Blog	#1 Votes	Total Votes	Total Points
1	Yellow Bricks	158	525	4191
2	Virtual Geek	111	379	2938
3	Scott Lowe	56	397	2889
4	NTPro.nl	22	349	2062
5	RTFM Education	7	305	1734
6	Virtualization Evangelist	13	286	1482
7	VM/ETC	5	230	1138
8	Gabe's Virtual World	8	233	1096
9	Virtual Storage Guy	58	153	990
10	Virtu-al	18	160	831
11	Virtualization Pro	3	152	744
12	vCritical	15	144	738
13	VMware Tips	5	155	726
14	Frank Denneman	22	112	697
15	The VM Guy	2	148	643
16	Planet VM	4	136	633
17	The SLOG	8	122	614
18	VMGuru.nl	18	112	569
19	Mike D's	2	122	537
20	Hypervizor	8	98	453
21	TechHead	12	81	452
22	vReference	11	90	443
23	Pivot Point	2	82	380
24	Technodrone	3	84	378
25	Chris Wolf	0	87	375
26	By The Bell	1	82	358
27	Ken's Virtual Reality	1	79	347
28	Vinternals	4	80	314
29	Arnim van Lieshout	9	52	304
30	Run Virtual	0	63	287
31	View Yonder	0	56	276
32	ESX Virtualization	1	53	263
33	VMware Info	9	56	261
34	Professional VMware	2	57	243
35	Everyday Virtualization	8	48	231
36	IT 2.0	3	46	212
37	Virtual Future	8	42	209
38	2 VCPs	1	53	207
39	Virtual Kenneth	5	34	200
40	Gerben Blog	1	33	194
41	VM Today	1	41	189
42	The Virtualization Practice	6	38	188
43	Andi Mann	0	40	188
44	SearchServerVirtualization	4	43	187
45	Virtual Insanity	4	40	180
46	Virtual VCP	0	41	179
47	Musings of Rodos	4	35	176
48	Virtualized Reality	9	31	176
49	Beaver's Virtual Dam	2	42	176
50	Jume	1	38	175
51	Rational Survivability	0	40	174
52	Tech Prosaic	3	33	174
53	Vinf.net	2	30	171
54	VMPros.nl	2	47	166
55	Jase's Place	0	37	162
56	Virtual Lifestyle	0	32	160
57	Virtualization Information	1	31	140
58	Virtual Cloud	0	31	133
59	Roger Lund	2	28	123
60	Lone SysAdmin	0	31	115
61	Malaysia VM	2	29	114
62	Daily Hypervisor	0	27	111
63	The HyperAdvisor	0	24	86
64	Going Virtual	0	6	38
65	Knudt Blog	0	8	28
66	LucD.info	0	4	23

I was commissioned by Veeam to write a white paper for them to coincide with their new Veeam 4.0 product launch. I was named as one of the 300 vExperts by VMware in Fed. 2009. I attended VMworld this year as a member of the press and was again a judge for the Best of VMworld awards in the Security category. Besides all that I still find time to maintain this website, sorting through links and collecting and posting them is very time-consuming. Below is a list of some of work of mine that was published in 2009:

• Everything you need to know about vSphere and data storage (SearchStorage.com magazine)
• VMware 2009: Never a dull moment (SearchVMware.com)
• The Mechanics of VMware GO (SearchVMware.com)
• A review of VMware disk-to-disk backup apps: Veeam, Vizioncore, PHD Virtual and VDR (SearchDataBackup.com)
• Virtual-to-physical conversions: Why and how (SearchVMware.com)
• VMware vShield Zones: What it is and how it works (vShield Zones Pt. 1) (SearchVMware.com)
• Installing and configuring vShield Zones (vShield Zones Pt. 2) (SearchVMware.com)
• Quick tips for managing vShield Zones (vShield Zones Pt. 3) (SearchVMware.com)
• Determining the right size for your VMware VMFS data stores (SearchVMware.com)
• Virtualizing may cost more, but the benefits are worth it (SearchVMware.com)
• Choosing a virtual disk type for VMware ESX: Raw, thick and thin disks (SearchVMware.com)
• VMware vSphere: Got 64-bit hardware? (SearchVMware.com)
• Blades vs. rack servers: Basic purchasing principles still apply (Choosing hardware for virtual servers series Pt. 1) (SearchServerVirtualization.com)
• Selecting CPU, processors and memory for virtualized environments (Choosing hardware for virtual servers series Pt. 2) (SearchServerVirtualization.com)
• PowerShell scripting with VMware ESX tutorial: Installing and using the PowerShell tool (PowerShell series Pt. 1) (SearchVMware.com)
• PowerShell scripting for VMware ESX: Using Quest’s PowerGUI PowerPack script editor (PowerShell series Pt. 2) (SearchVMware.com)
• What is Changed Block Tracking in vSphere? (Virtualization Pro blog)
• Does virtualization really reduce administrator headcount and workloads? (Virtualization Pro blog)
• Several ways to explain virtualization to a non-techie person (Virtualization Pro blog)
• What are IP Pools and vApps? (Virtualization Pro blog)
• What is virtualization? (Virtualization Pro blog)
• Removing old hardware after a P2V conversion (Virtualization Pro blog)
• Are you taking advantage of VMware knowledge on the Web? (Virtualization Pro blog)
• Virtualizing Lotus Domino servers: Support concerns (Virtualization Pro blog)
• The process behind judging the Best of VMworld awards (Virtualization Pro blog)
• Using Storage VMotion to keep critical servers up when shared storage is down (Virtualization Pro blog)
• Scheduling virtual CPUs (Virtualization Pro blog)
• More details on VMware’s Fault Tolerance feature (Virtualization Pro blog)
• The key to successfully virtualized Domino servers (Virtualization Pro blog)
• VMware’s hardware compatibility list (Virtualization Pro blog)
• VMware Fault Tolerance: What it is and how it works (Virtualization Pro blog)
• Pointers for using thin-provisioned disks (Virtualization Pro blog)
• Troubleshooting mysterious VM resets (Virtualization Pro blog)
• What it means to be a vExpert (Virtualization Pro blog)
• Virtualizing Lotus Domino servers (Virtualization Pro blog)
• Lord of the virtual rings: Why hardware matters to your VMs (Virtualization Pro blog)
• A brief history of VMware (Virtualization Pro blog)
• Defragmenting virtual machine disk files (Virtualization Pro blog)

Can anyone knock Duncan Epping from the top spot? Chad Sakac made a run up to the #3 position in the last vote, can he go any higher? Interesting enough, Scott Lowe in the #2 position now works for Chad Sakac, can Scott stay ahead of his new boss? There are also many new bloggers out there producing some great content who could make the list like Hany Michael, Scott Sauer/Aaron Sweemer, Maish Saidel-Keesing, Scott Drummonds, Frank Denneman and Steve Chambers. Your votes will decide the winners and help decide the Top 25 bloggers. I’ve decided to expand the Top 20 to the Top 25 because there are so many great blogs out there.

So here’s how it works, you pick your top 10 favorite blogs, the survey will be open for two weeks. Once all the votes have been cast I’ll add them up and using weights (#1 vote = 10 points, #2 vote equals 9 points, etc.) and calculate the Top 25 blogs. Because of the cost of the online survey tool I’ve acquired a sponsor to help cover the costs. TrainSignal is the sponsor of this survey and as an added bonus they are giving away two copies of their awesome vSphere DVD training course to two voters who will be randomly selected. to vote you simply number your 10 favorite blogs from 1 to 10 with 1 being your most favorite blog. Your numbers must add up to 55 (1 through 10 added together) and we have controls in place so you can only vote once. The top 10 winners will receive a graphic image with their place number that they can display on their website.

When casting your vote we hope you are fair and use the following guidelines:

• frequency (how often they post)
• quality (how well the posts are)
• length (how long the posts are)
• longevity (how long they have been blogging for)

Please note for added protection this time against duplicate votes I’m now using IP address protection which limits one vote per IP address. While this method is more effective then using cookies it may prevent multiple users in corporate environments that use NAT from casting votes. As a workaround to that situation you can have people vote from a mobile device or home computer instead.

So bloggers, start your engines and let the voting begin, you can go to http://vote.vsphere-land.com or click the below link to cast your vote…

Click here to vote