Colocation to Virtualization

Will Be Back Soon

Kevin Goodman — Mon, 26 Aug 2013 19:50:17 +0000

Definitely need to get back to blogging. Work has been crazy over the past year. Been doing a lot of Cisco UCS, EMC VNX, and Cisco VoIP. More to come soon

– Kevin

Filed under: Uncategorized

Linux IPtables Port Redirect

Kevin Goodman — Mon, 16 Apr 2012 19:42:56 +0000

So a few days ago we needed to do a port redirect on a server. This was a new implementation of an app server, but the link for all the clients refered to port 80. The new install listed on port 8080.

A few ideas were kicked around like installing apache to listen and do a metaredirect to the new location. Then it clicked, why not use IPtables?

Below is how we redirected port 80 to 8080

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080

Once completed and tested, I saved the config as follows

iptables-save > /etc/sysconfig/iptables

The above is for a RedHat Linux server. IPtables pulls its config from /etc/sysconfig/iptables file on startup.

Filed under: Linux, Networking

Cisco UCS B200-M1 Interaction Times 1.3(1c)

Kevin Goodman — Tue, 13 Dec 2011 16:50:23 +0000

So I wrote this up a long time ago and just never posted it. I have a lot more UCS stats saved on text files that I should be able to dig up. I completely forgot about this until an email yesterday.

So when it comes to maintenance, I time almost everything. I really like being able to estimate how long specific actions should take to complete. That is what this post is all about. I will be going over startup times, dissassociation, and association of service profiles on Cisco B200 M1 blades. The test was done using blades having different RAM configurations across multiple chassis.

Also to note, our blades boot from SAN to a fiber connected EMC CX4-240 Clariion. The storage on the CX4 supporting these and 10 other boot from SAN blades LUNs is as follows:

Qty 4 – 600gb 10kRPM disks setup in a RAID 10 group.

The following firmware versions are the same between both blades being tested:

UCS Manager:	1.3(1c)
Blade CIMC Controller:	1.3(1c)
Blade BIOS:	S5500.1.3.1c.0.052020101544
Blade Interface Card:	1.3(1c)

Storage path: Palo interface -> fiber -> Cisco Nexus 6120 -> fiber -> EMC CX4

Test 1 Consists of a B200-M1 blade with the following configuration

OS:	VMware vSphere 4
CPUs:	Dual quad core Intel Xeon X5570 2.93GHz
RAM:	98 gig
Interface Card:	Cisco UCS VIC M81KR Virtual Interface Card (Palo)
Boot:	FCoE to EMC CX4-240 SAN

Bootup and shutdown:

Shutdown from running:	~48 seconds
Power-off to ESX Grub boot loader:	~2.27
Power-off to ESX console:	~4.15

Service Profile associations:

Power-off to completed dissassociation:	~3.08
Dissassociatd to completed association:	~3.30

Test 2 Consists of a B200-M1 blade with the following configuration

OS:	Windows 2008
CPUs:	Dual quad core Intel Xeon X5570 2.93GHz
RAM:	24 gig
Interface Card:	Cisco UCS VIC M81KR Virtual Interface Card (Palo)
Boot:	FCoE to EMC CX4-240 SAN

Bootup and shutdown:

Shutdown from running:	~58 seconds
Power-off to Windows loading bar:	~2.50
Power-off to Windows 2008 login console:	~3.77

Service Profile associations:

Power-off to completed dissassociation:	~3.09
Dissassociated to completed association:	~3.10

Note(s): The testing was done with blades that were already at the service profiles required firmware level. More time would have to be added if an firmware change was needed. Also, boot times will be dependent on the IOPs achievable to your storage system.

Filed under: Cisco UCS

Nimble Storage CS220G ISCSI Array Initial Install

Kevin Goodman — Fri, 05 Aug 2011 12:00:49 +0000

We received our Nimble Storage CS220G earlier this week. We are hoping for the device to fall into a tier between our EMC CX4 and standard NAS storage systems. Being that we are mostly 10gb at our datacenter, it was a definite requirement. Most other appliances around the same price point did not have any type of storage tiering, so I am also excited about that.

General Specs:

Dual controllers
RAW 12tb
Cache (SSD) RAW 320gb
Quad 10gbE connectivity (2 per controller)

Below shows the Nimble device being un-boxed. I highlighted the SSD drives in green

Nimble Un-boxing

Even though the unit is 3ru in size, they were able to stuff two controllers (heads) into it. Primary is highlighted blue and secondary green.

A small gallery including the racked unit. Notice it is in good company!

Nimble Controllers

" data-medium-file="https://colovirt.files.wordpress.com/2011/08/img_1652.jpg?w=300" data-large-file="https://colovirt.files.wordpress.com/2011/08/img_1652.jpg?w=497" />
Filed under: SAN (Storage Area Network)

RamSan 500 Pictures / Failure

Kevin Goodman — Mon, 07 Mar 2011 13:00:00 +0000

We had two DRAM cache modules fail in our RamSan 500 recently. The actual error was:

Uncorrected ECC event detected on boards 4,6

The DRAM cache fronts the Flash storage drives in the unit. Basically, loosing these two boards took down the unit completely. To get the unit back up, boards 4 and 6 needed to be removed. RamSan actually had me remove 4 of the boards instead of just the two that were bad. Guessing it has something to do with not being able to have slots not filled between active modules.
Doing so took the DRAM cache from 64 gigs down to 32. All data was lost on the FLASH RAID array, making us have to restore the Oracle database from backup. After the cards were removed, a low level format of the RAID array was done, the unit was operational again. Below are pictures from within the RamSan 500 for all the storage people out there.

Best part of this is we had racked two RamSan 630’s that were being configured as a mirrored pair. The plan was to implement them in two weeks. When the 500 failed, they were implemented that night.

Filed under: Datacenter, Hardware, RamSan, SAN (Storage Area Network)

Cisco Nexus 1000v 4.2 Upgrade Problems And Fix

Kevin Goodman — Tue, 01 Mar 2011 13:33:41 +0000

Update

I’m glad Jeff (see comments) caught where I went wrong. After his comment I went back through the n1000v_upgrade_software PDF from cisco. For version 4.0(4)SV1(3a), the VEMs should have been upgraded first. It is on page 9 of the document.

Upgrading from Release 4.0(4)SV1(3, 3a, or 3b) to Release 4.2(1)SV1(4)

Step 1 Upgrading the VEMs: Release 4.0(4)SV1(2, 3, 3a, 3b) to Release 4.2(1)SV1(4), page 20

Step 2 Upgrading the VSMs to Release 4.2(1)SV1(4) Using the Upgrade Application, page 33

I was following lower down in the document that showed an upgrade of the VSM first then the VEM. Looking closer that was for one that required an intermediate upgrade between the current running and the newer 4.2(1)SV1(4) firmware. Thanks for the catch!

Original post:

So last weekend was our Cisco Nexus 1000v upgrade. We were upgrading from 4.0(4)SV1(3a) to 4.2(1)SV1(4). What should have been an easy upgrade really turned into a huge headache. Below is a walk-through of the upgrade process with notes on where things went wrong.

To start, lets verify the current running state of the VSM (1000v switch) and the VEM (ESX host modules). Below you will see that they are all on version 4.0(4)SV1(3a). Also note the standby VSM module number is 2. This will be the first one reloaded.

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
2    0      Virtual Supervisor Module        Nexus1000V         ha-standby
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok
5    248    Virtual Ethernet Module          NA                 ok
6    248    Virtual Ethernet Module          NA                 ok
7    248    Virtual Ethernet Module          NA                 ok
8    248    Virtual Ethernet Module          NA                 ok
9    248    Virtual Ethernet Module          NA                 ok

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.0(4)SV1(3a)      0.0
2    4.0(4)SV1(3a)      0.0
3    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
4    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
5    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
6    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
7    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
8    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
9    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)

Show boot displays the current location/firmware being used on bootup. This will be changed with the upgrade.

1000vSW# sh boot
sup-1
kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin
system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin
sup-2
kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin
system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin
No module boot variable set

Just like any other devices, make sure there is enough room on the bootflash device to store the new firmware

1000vSW# dir

Usage for bootflash://
  481083392 bytes used
 1907187712 bytes free
 2388271104 bytes total

After adding up the file size of the both files, there is enough on the bootflash to hold the new firmware. I already had the files on a host running scp, but the files can be loaded from any hosts running the following protocols

TFTP

FTP

SCP

SFTP
The syntax for copying over SCP is as follows

copy scp://user01@127.0.0.15/IOS/1000v/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin bootflash://
copy scp://user01@127.0.0.15/IOS/1000v/nexus-1000v-mz.4.2.1.SV1.4.bin bootflash://

From the active VSM (module 1 in this case), the new firmware is loaded into the configuration via the “install” command.

1000vSW# install all system bootflash:nexus-1000v-mz.4.2.1.SV1.4.bin kickstart bootflash:nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin
System image sync to standby is in progress...
System image is synced to standby.
Kickstart image sync to Standby is in progress...
Kickstart image is synced to standby.
Boot variables are updated to running configuration.

The boot variables were updated per the above, but it is still worth verifying. Below shows that the running configuration was updated to load the new firmware.

1000vSW# sh running-config | inc boot
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-1
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-2
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-2

I almost skipped this command the first time. Make sure to save the configuration to startup-config or the reload of standby module will load up the startup-configs boot variables. That would bring it up on the same 4.0(4)SV1(3a) version.

1000vSW# copy running-config startup-config
[########################################] 100%

We saw the module status earlier in this post, but I will reiterate it here. Module 2 shows as the standby VSM here, so it will be reloaded first.

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
2    0      Virtual Supervisor Module        Nexus1000V         ha-standby

Time to reboot the standby module. This is done as follows.

1000vSW# reload module 2
This command will reboot standby supervisor module. (y/n)?  [n] y
about to reset standby sup
1000vSW# 2011 Feb 25 19:39:33 1000vSW %PLATFORM-2-PFM_MODULE_RESET: Manual restart of Module 2 from Command Line Interface

Module 2 is still restarting, notice it missing under “Mod”

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok

* you can bring up the console of the standby switch in vCenter to check progress

Ok, this is where everything went wrong for us. A typical upgrade would have the module connect back in showing the new version of code running on it. For us, this did not happen. Instead module 1 and 2 were not able to communicated with each other. Both decided to become active.

This became a big problem. All VMs running were still able to pass traffic across the network, but none of ESX hosts showed up as modules in the VSM, thus isolating them. First symptom I saw was connectivity loss via SSH to the VSM IP. It seemed to happen about every two minutes. luckily I saw the firmware version changing each time I did a show module after reconnecting. So the two 1000v modules were in IP conflict with each other, stealing it away from one another every few minutes.

I waited for the module running version 4.0(4)SV1(3a) to grab the IP again and issued a reload. This was module 1. Once it restarted and came up on the new firmware, it started talking with module 2.

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          ha-standby
2    0      Virtual Supervisor Module         Nexus1000V          active *

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Now that we are on the same firmware, it was time to make module 1 primary/active once more. This was non-intrusive.

1000vSW# system switchover

Once more to verify that module 1 has become active:

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Also to be safe, have the VSM reconnect with Virtual Center.

1000vSW## configure t
Enter configuration commands, one per line.  End with CNTL/Z.
1000vSW#(config)# svs connection VirtualCenter
1000vSW#(config-svs-conn)# connect
1000vSW#(config-svs-conn)# end

Verify the connection as well.

1000vSW#  show svs connections

connection vcenter:
    ip address: 127.0.0.99
    remote port: 80
    protocol: vmware-vim https
    certificate: default
    datacenter name: Test
    DVS uuid: 55 55 55 55 55 55 55 55
    config status: Enabled
    operational status: Connected
    sync status: Complete
    version: VMware vCenter Server

At this point we should be good. At least that is what I thought. Going back to what I said before
“but none of ESX hosts showed up as modules in the VSM.”
Yeah, I didn’t notice that untill after the fact. Normally I vmotion over an individual host to a different server to test connectivity, but this time I didn’t.

I selected our first vSphere (ESX) host and entered maintenance mode, thus vmotioning all VMs off to other hosts. Immediately they fell off of the network. All pings were lost. These were running critical/production servers. Of course we were in a maintenance window, but this still was worst case scenario. Normally each hosts VEM would show up in the show module, but they did not after the upgrade.

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-00-00-00-00-00 to 00-00-00-00-00-00  NA
2    00-00-00-00-00-00 to 00-00-00-00-00-00  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    127.0.0.55     NA                                    NA
2    127.0.0.55     NA                                    NA

The above resembled what I was seeing. Just the VSM modules. The mac addresses were correct, I just changed them to all 0’s here. We tried shutting down a VM and moving it to different hosts in the cluster, but still no pings. I also tried stopping and starting the VEM via command line on the ESX host.

The only thing that fixed this was pushing the VEM upgrades. This was done using VMware Update Manager. Very easy, but we had to put the hosts into maintenance mode to do so. This wound up being very time-consuming. Once server 1’s VEM was updated, we were able to vMotion VMs to it without losing network connectivity to them. It went like this:

Update ESX host 1 VEM to 4.2

vMotion ESX host 2 VMs to host 1

Update ESX host 2 VEM to 4.2

vMotion ESX host 3 VMs to host 2

Update ESX host 3 VEM to 4.2

vMotion ESX host 4 VMs to host 3

Update ESX host 4 VEM to 4.2

Repeat until done!

It would have been a lot easier if the hosts could have been placed into maintenance mode and VMs automatically migrated to other members in the cluster. Once completed, everything looked fine in the VSM. Even the hosts showed up!

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    127.0.0.55       NA                                    NA
2    127.0.0.55       NA                                    NA
3    127.0.0.24       22222222-2222-2222-2222-22222222221g  vm_host01 
4    127.0.0.26       22222222-2222-2222-2222-22222222222g  vm_host03 
5    127.0.0.27       22222222-2222-2222-2222-22222222223g  vm_host04 
6    127.0.0.28       22222222-2222-2222-2222-22222222224g  vm_host05 
7    127.0.0.25       22222222-2222-2222-2222-22222222225g  vm_host02 
8    127.0.0.29       22222222-2222-2222-2222-22222222226g  vm_host06
9    127.0.0.120      22222222-2222-2222-2222-22222222228w  vm_host07

A new rule for us to now follow: Always put one host into maintenance mode before upgrading!
This would have allowed us to upgrade without downtime in this scenario.

Filed under: Cisco UCS, Networking, VMWare

10gb Ethernet – A Year Later / Buy It Carefully

Kevin Goodman — Wed, 16 Feb 2011 13:00:27 +0000

What I am getting at here is do not underestimate the amount of ports you will be using. We have implemented Cisco UCS with 6120 switches but also installed a pair of Nexus 5010 switches as well. Going in, we had only planned for a few devices to be connected to the 10gb Ethernet network outside of the UCS infrastructure. Looks like we underestimated!

Once we got over the fear of FCoE reliability, almost everything is now being ordered with 10gbE. The only reason of worry was during this implementation over a year ago, FCoE was not in the main stream yet. It was almost “bleeding edge” in the market.

Where are we a year later?

We are expanding our Cisco UCS environment to at least an additional chassis

EMC CX-4 has 10gbE fiber modules for ISCSI (instead of RDMs for some VMs)

DataDomain 670 is 10gbE fiber connected

EMC NX4 NAS has 10gbE fiber connectivity

We are planning a purchase of either a CX480 or VNX 5700 which will be 10gb FCoE

Hoping to order a pair of Cisco Nexus 7000s

One of the big killer of ports in our environment is having to tie the Gigabit Ethernet switches into them. This is wasting 4 ports that could be 10gb pre switch. Hopefully that will be resolved with a future purchase of Nexus 7000 switches. We also have a pair of fabric extenders connected into the 5010s. This also takes up two 10gbE ports per 5010.

Below is a port listing from one of our Nexus 5010 switches. This is after we removed 3 ports (per switch) from physical servers that were tied into the 10gbE infrastructure. As you can see from below, we only have 5 ports per switch left (10 total if you count the redundant switch)! That is only 5 more devices at the most.

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        x     eth  trunk  up      none                       1000(D) 1
Eth1/2        x     eth  trunk  up      none                       1000(D) 1
Eth1/3        x     eth  trunk  up      none                       1000(D) 1
Eth1/4        x     eth  trunk  up      none                       1000(D) 1
Eth1/5        x     eth  trunk  up      none                        10G(D) --
Eth1/6        x     eth  trunk  up      none                        10G(D) --
Eth1/7        x     eth  access up      none                        10G(D) --
Eth1/8        x     eth  access up      none                        10G(D) --
Eth1/9        x     eth  trunk  down    Link not connected          10G(D) --
Eth1/10       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/11       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/12       x     eth  access down    SFP not inserted            10G(D) --
Eth1/13       x     eth  access up      none                        10G(D) --
Eth1/14       x     eth  access down    SFP not inserted            10G(D) --
Eth1/15       x     eth  trunk  up      none                        10G(D) --
Eth1/16       x     eth  trunk  up      none                        10G(D) --
Eth1/17       x     eth  fabric up      none                        10G(D) --
Eth1/18       x     eth  fabric up      none                        10G(D) --
Eth1/19       x     eth  trunk  up      none                        10G(D) 2
Eth1/20       x     eth  trunk  up      none                        10G(D) 2

At the 6120’s end (UCS switches) we are fine. Currently have 14 ports on each switch available. That should give us the ability to wire in 7 more chassis with 2 – 10gbE uplinks per IOM. A total of 4 links / 40gb throughput per chassis. Our CIFS and NFS access is a lot faster running through the EMC NX4’s 10gbE.

So if you are looking into 10gbE for our network, it would help to have at least a rough draft of where you want your datacenter to be a few years from now. Pretty much all major storage companies are doing some sort of 10gbE connectivity. Using it makes administration a lot easier. FCoE simplifies things even more, including eliminating the need for separate fiber switches. To sum it up, I love 10gbE and would recommend it even if not planning on incorporating FCoE or UCS in your environment.

Filed under: Cisco UCS, Data Domain, Datacenter, deduplication, EMC, NAS, SAN (Storage Area Network)

EMC NX4 Expansion

Kevin Goodman — Thu, 10 Feb 2011 13:00:24 +0000

So our EMC NX4(s) are starting to actually look like a real storage system. We have two units that were installed a few months ago with 1 tray of drives. We ordered 4 more DAEs (2 for each sytem) for needed expansion. Should have them up and configured shortly.

So can someone explain to me why EMC sells systems with non-matching face plates. Like our RecoverPoint systems attached to our CX4 actually have CX3 style faceplates. Same with the NX4, the 3 1RU components at the bottom faceplates are completely different than the rest. Either way, it still is a nice system.

Filed under: Cisco UCS, Datacenter, EMC, Hardware, SAN (Storage Area Network)

EMC DataDomain 670 Implementation Pictures

Kevin Goodman — Mon, 24 Jan 2011 12:00:31 +0000

DD 670 Front

We received our new DataDomain 670 last week and it has definitely had a face lift from the 500 series. First, it is a 2u versus a 3u unit. Also, the 600 series no longer have the annoying third power supply. The drives are also horizontal instead of vertical. Even slots in the back are different. I wish we had a multi-shelf unit one. Below are a few pictures from the unpacking and racking.

Notes: Only complaint is the rails. I really wish they rails on the server were pegged style – drop into extended rack rail types (IBM). Instead, the rails on the servers are fixed and have to be lined up perfectly with the ones in the rack. They then slide straight into the rack rails and lock. I find it a lot hard with this kind of rails to rack heavy equipment alone.

Filed under: Colocation, Data Domain, Datacenter, deduplication, NAS

Cisco UCS 6120 Switch Replacement

Kevin Goodman — Tue, 18 Jan 2011 12:00:54 +0000

So back in August of 2010 I had to do a 6120 switch replacement, due to a bad port on the switch. I know I commented on twitter during the process but did not here.

To be safe, we were in a maintenance window, but servers were still up and running while the switch replacement was being done.

A little more information on our setup

UCS Manager was running at v1.3(1c)

All blades boot from SAN via FCoE

Cisco MDS 9124s sits between the 6120s and SAN Storage (EMC)

RedHat Linux blades were left running

Windows 2008 blade was left running

VMware vSphere blades were left running

All blades contain the VIC N81kr (PALO) interfaces

Two UCS chassis in the environment

In “theory” UCS has a lot of redundant built-in, but I had yet to test out taking down a switch. I had at this point pulled IOM modules, blades, SFP cables from an active chassis but not a 6120.
Just to give you an idea of how many dropped packets were lost during the process, below are pings that were done during the removal.

Pings to a vSphere host in Cisco UCS

Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63

Ping statistics for 127.0.0.59:
    Packets: Sent = 2413, Received = 2410, Lost = 3 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 75ms, Average = 0ms

Pings to the cluster IP of 6120s

Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
Reply from 127.0.0.41: bytes=32 time=4ms TTL=63
Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63

Ping statistics for 127.0.0.41:
    Packets: Sent = 2339, Received = 2316, Lost = 23 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 8ms, Average = 1ms

So all servers were fine. Nothing went down and no data was lost to the storage going to the SAN. It took about 1 hour to replace switch which entailed:

Un-rack bad 6120

Rack replacement 6120

Upgrade firmware on the new 6120. It was sent to us with version 1.2

Join new 6120 to cluster

Note(s): Wish that I would have noticed the replacement switch came with an older firmware. I spent a decent amount of time setting it up on a temporary IP (standalone), pushing new firmware to it, and then updating it. Once done, I had to wipe the config and link it to the other 6120 and join the cluster.

Filed under: Cisco UCS, Datacenter, Networking

TMS RamSan 630 Firmware Upgrade

Kevin Goodman — Mon, 17 Jan 2011 18:28:04 +0000

So I have completed the first two of our RamSan firmware updates. Turns out (surprisingly) that the process is very easy! Instead of fluffing this post up a lot with text, I am just posting the pictures in order as they occurred during the upgrade. If you have any questions, just comment or email me.

Filed under: RamSan, SAN (Storage Area Network)

Cisco UCS CLI – Service Profile Associations

Kevin Goodman — Thu, 09 Dec 2010 20:01:54 +0000

I am a very big fan of using CLI interfaces. I have had to bounce interfaces and restart servers at times when remote connectivity was really slow (3g, Edge, tethered, etc). I love the UCS Manager GUI, but it is definitely bulky and graphically more intensive than the CLI. That is where this post comes in.

Below is a run through of associating and disassociating a service profile from one blade to the other.

To begin with, lets find out the state of all service profiles on the system.

ucsm01-A# show service-profile assoc
Service Profile Name Association    Server Server Pool
-------------------- -------------- ------ -----------
db05           Associated     2/3
vcb01          Unassociated
vm02           Associated     1/4
vm03           Associated     1/5
vm04           Associated     2/5
vm05           Associated     2/6
esxi01         Unassociated

For testing, I am going to use “vcb01”. The include function is used below to only show me the stats of the vcb01 service profile.

ucsm01-A# show service-profile assoc | include vcb
vcb01          Unassociated

There are two ways to access the service profile. The first method is to find the UUID.

ucsm01-A# show service-profile identity name vcb01
Service Profile Name: vcb01
UUID Suffix Pool: UUID_Pool
Dynamic UUID: 11000000-0000-0000-2100-00000000148f

    VNIC FC Node:
    WWNN Pool: Blade_WWNN
    Dynamic WWNN: 20:11:00:25:B5:00:00:2F

Since we will be associating a profile with a blade, make sure that it is currently available

ucsm01-A# show server assoc
Server  Association  Service Profile
------- ------------ ---------------
1/1     None
1/3     None
1/4     Associated   vm02

Since we know the uuid, “scope” is used to modify its attributes.

ucsm01-A# scope service-profile dynamic-uuid 11000000-0000-0000-2100-00000000148f

Once we are in, associating a server is very easy.

ucsm01-A /org/service-profile # associate server 1/3

Before the above action takes place, “commit-buffer” must be issued. This is identified by the “*” before the “#”

ucsm01-A /org/service-profile* # commit-buffer

Time to check on the status of the association.

ucsm01-A /org/service-profile # show fsm status

Service Profile Name: vcb01

    FSM 1:
        Remote Result: Not Applicable
        Remote Error Code: None
        Remote Error Description:
        Status: Associate Nic Config Pnuos
        Previous Status: Associate Nic Config Pnuos
        Timestamp: 2010-11-19T13:53:15.093
        Try: 1
        Flags: 0
        Progress (%): 32
        Current Task: Configure adapter for pre-boot environment on server 1/3(FSM-STAGE:sam:dme:ComputeBladeAssociate:NicConf
igPnuOS)

Checking for the last time shows progress as 100%.

ucsm01-A /org/service-profile # show fsm status

Service Profile Name: vcb01

    FSM 1:
        Remote Result: Not Applicable
        Remote Error Code: None
        Remote Error Description:
        Status: 0
        Previous Status: Turnup Success
        Timestamp: 2010-11-19T13:56:56.975
        Try: 0
        Flags: 0
        Progress (%): 100
        Current Task:

Make sure that the server was booted

ucsm01-A /org/service-profile # power up
ucsm01-A /org/service-profile* # commit-buffer

To exit the service profile scope, just type “end”.

ucsm01-A /org/service-profile # end

Another way to access the service profile without specifying the UUID is as follows.

First, you have to find out the organization setup in the UCS config.

ucsm01-A# show org

Organizations:
    Name
    ----
    / (root)

Command sytax:

ucsm01-A# scope service-profile ?
  dynamic-uuid  Dynamic UUID
  org           Org
  server        Server

Knowing the org, the syntax changes to the following:

ucsm01-A# scope service-profile org / vcb01

This method is a little faster, but make sure you do not fat finger hostnames if they are similar.

Disassociation is just as easy. Once in the service profile scope:

ucsm01-A /org/service-profile # disassociate
ucsm01-A /org/service-profile* # commit-buffer

Filed under: Cisco UCS

Cisco UCS B200-M1 Interaction Times

Kevin Goodman — Thu, 18 Nov 2010 06:15:16 +0000

So when it comes to maintenance, I time almost everything. I really like being able to estimate how long specific actions should take to complete. That is what this post is all about. I will be going over startup times, dissassociation, and association of service profiles on Cisco B200 M1 blades. The testing was done using blades having different RAM configurations across multiple chassis.

Also to note, the tests were done with blades using FCoE boot from SAN to a fiber connected EMC CX4-240 Clariion. The storage on the CX4 supporting these and 10 other boot from SAN blades LUNs is as follows:

Qty 4 – 600gb 10kRPM disks setup in a RAID 10 group.

The following firmware versions are the same between both blades being tested:

UCS Manager:	1.3(1c)
Blade CIMC Controller:	1.3(1c)
Blade BIOS:	S5500.1.3.1c.0.052020101544
Blade Interface Card:	1.3(1c)

Storage path: FCoE through Palo interface -> Cisco Nexus 6120 -> fiber -> EMC CX4

Test 1 Consists of a B200-M1 blade with the following configuration

OS:	VMware vSphere 4
CPUs:	Dual quad core Intel Xeon X5570 2.93GHz
RAM:	98 gig
Interface Card:	Cisco UCS VIC M81KR Virtual Interface Card (Palo)
Boot:	FCoE to EMC CX4-240 SAN

Bootup and shutdown:

Shutdown from running:	~48 seconds
Power-off to ESX Grub boot loader:	~2.27
Power-off to ESX console:	~4.15

Service Profile associations:

Power-off to completed dissassociation:	~3.08
Dissassociatd to completed association:	~3.30

Test 2 consists of a B200-M1 blade with the following configuration

OS:	Windows 2008
CPUs:	Dual quad core Intel Xeon X5570 2.93GHz
RAM:	24 gig
Interface Card:	Cisco UCS VIC M81KR Virtual Interface Card (Palo)
Boot:	FCoE to EMC CX4-240 SAN

Bootup and shutdown:

Shutdown from running:	~58 seconds
Power-off to Windows loading bar:	~2.50
Power-off to Windows 2008 login console:	~3.77

Service Profile associations:

Power-off to completed dissassociation:	~3.09
Dissassociated to completed association:	~3.10

Note(s): The testing was done with blades that were already at the service profiles required firmware level. More time would have to be added if a firmware change was needed. Also, boot times will be dependent on the IOPs achievalbe to your storage sytem.

Filed under: Cisco UCS, EMC, Linux, SAN (Storage Area Network), VMWare

RamSan 630 (SSD) Racking And Flash Module Installation

Kevin Goodman — Tue, 02 Nov 2010 12:49:08 +0000

We are continuing to roll out Texas Memory RamSan SSD devices to support our Oracle environments. This time we went with a RamSan 630 for our disaster recover location to host our primary Oracle databases at this site.

Some specs on the system:

Capacity:	Up to 10 TB
Storage Medium:	SLC Flash
Maximum Bandwidth:	10 GB/s
Read IOPS:	1 M IOPS
Read Latency:	250 microseconds
Write IOPS:	1 M IOPS
Write Latency:	80 microseconds
Expansion Slots:	5 slots
Fibre Ports per Card:	2 ports/card
Fibre Speed per Port:	4 Gb/s
Power Consumption:	500 W
Dimensions:	5.25″ (3U) x 20″
Form Factor:	3U
Weight:	55 lbs

RamSan Box

Turns out that the hot-spare flash module was shipped separately so I had to install that myself. Pictures from the racking and module installation is below.

Filed under: Datacenter, Hardware, RamSan, SAN (Storage Area Network)

Pictures From Datacenter NAS Refresh (EMC Celerra NX4 and NetApp 3020)

Kevin Goodman — Wed, 27 Oct 2010 13:00:49 +0000

So once again I found myself at the datacenter shuffling around equipment. These pictures and videos are from our secondary (DR) location. I was removing our older single headed NetApp FAS3020 and replacing it with a new EMC NX4 system. Difference here from the “main/production” site is the NX4 there is dual headed and is also 10 gigabit Ethernet attached.

EMC NX4 Racked Front

" data-medium-file="https://colovirt.files.wordpress.com/2010/10/sany0051.jpg?w=300" data-large-file="https://colovirt.files.wordpress.com/2010/10/sany0051.jpg?w=497" />

Below are a few videos of the equipment. Be careful with the sound! These were taken in the datacenter and you will hear the typical rumbling of servers and AC units.

Video of the NetApp FAS 3020 once removed

Video of the EMC NX4 waiting to go in

Filed under: Colocation, Datacenter, EMC, Hardware, NAS, SAN (Storage Area Network)