Image via Wikipedia

This was a less than pleasant problem to figure out. However should you experience these problems, let this be one more place to look.

I decided it was time to take my VoIP servers and make them redundant. I only had two NICs on each machine, using one for private traffic and the other for public. Since Citrix XenServer supports the bonding of nics and 802.1q vlan tagging, I thought that this would make a good combination for HA, in the event of a NIC or switch failure.

So away I went. Bonding is fairly simple in Citrix XenCenter. You just tell it you want to create a bond, select the nics and in a few seconds you have a bond. Great, just create the networks and vlan then life is all peachy.

I thought I was all good until the next day I started to receive calls that voice was choppy. Let me preface this with, the bonding wasn't the only thing I changed, but were focusing on it so I'll only make reference to it. After many packet captures and testing I found that packets were not being lost, they were being delayed by about three seconds.

At this point I started to investigate how citrix xenserver bonds their networks. It turns out they use the standard linux kernel modules to do the bonding. This is very good. The bad part is, they choose to only support 'balance-slb' which isn't well documented at all. (If you find documentation on this type of load balancing, please comment below.) The main idea with this type of load balancing is to share the load over both links. They do this by changing what interface is used ever 10-30 seconds. (I can't get a strait answer from Citrix on the time. One engineer said 10 the other said 30 seconds) Well that sounds all fine and dandy until you checkout the default settings for the bond:

# /opt/xensource/libexec/interface-reconfigure
# Line 862
bond_options = {
"mode": "balance-slb",
"miimon": "100",
"downdelay": "200",
"updelay": "31000",
"use_carrier": "1",
}

Ouch! Yes that is a 31000ms up-delay you see there, translated into seconds, thats 3.1 seconds. There it was, my 3 second delay. While the interface was coming up, the kernel was queuing all the packets, as soon as the interface was up it would send all the packets as fast as it could. The ICMP ping packets arrived out of order and the audio was dropped on phones.

Great now that I have identified the problem, how do I fix it? Back in my vmware days, yes I am a VI3 vmware certified professional (VCP), I learned about NIC teaming also known as 802.3ad Link aggregation. There are two types of link aggregation, link and dynamic. Dynamic is the best as it uses both links equally at the same time and provides frameless failover.

Great so now I decided to use a tried and true protocol how do I implement it? No problem. I use Brocade Fastiron CX switches, which are amazing. To use 802.3ad one must have a switch that supports the protocol. I logged into the switch found the two ports that were to be used and typed in

 link-agg active 

Now the last step is to get xenserver to use 802.3ad. After I read about the different methods to bond in linux http://sourceforge.net/projects/bonding/ I was off to modify citrix xenservers default config. They however want you to use a special method to configure the device, but I find this way the most painless and scriptable. Here it is:

# /opt/xensource/libexec/interface-reconfigure
# Line 862
# The bond option defaults
bond_options = {
"mode": "802.3ad",
"lacp_rate": "1",
"miimon": "100",
"downdelay": "200",
"use_carrier": "1",
}

After I saved the modification, I rebooted the servers and viola, 802.3ad Dynamic Link Aggregation up and working. I didn't even have to reconfigure my bonds. I ran some tests and of course this fixed the choppiness and everyone was happy again.

This will allow me to have true fail-over and the added bonus is, I can use the full 2gigs of bandwidth at any time, which on some of my other servers has come in handy since I'm using iSCSI, I'll talk about that in a future post.

UPDATE

There is a better way of doing this so it won't break any of your updates and you don't have to maintain custom code. You need set a parameter on your bond interface.

# Find the network name
xe network-list 
uuid ( RO)                : 22d8b51f-ec0c-16a8-47c7-5a42093bef37
          name-label ( RW): Public
    name-description ( RW): 
              bridge ( RO): xapi3
uuid ( RO)                : 1621b2ec-b549-cc8b-6a4c-839b5017a58d
          name-label ( RW): Guest installer network
    name-description ( RW): Network on which guests will get assigned a private local IP address
              bridge ( RO): xapi0
uuid ( RO)                : aa0dec75-2f96-9de2-4517-ef21bedacc79
          name-label ( RW): iSCSI (bond)
    name-description ( RW):
              bridge ( RO): xapi4
uuid ( RO)                : 237cdc19-3b84-0c97-7a98-842a97aa63e7
          name-label ( RW): Internal
    name-description ( RW):
              bridge ( RO): xapi5
uuid ( RO)                : 0e7530ad-7506-00bb-5d69-8975e4ecaf32
          name-label ( RW): iSCSI B
    name-description ( RW):
              bridge ( RO): xenbr3
uuid ( RO)                : 6316a5ea-be2c-c425-bf8c-20909c9c5aa7
          name-label ( RW): Main Network
    name-description ( RW): 
              bridge ( RO): xapi2
uuid ( RO)                : bf731568-91f5-1109-a6e8-f56d2b1a2ba9
          name-label ( RW): Pool-wide network associated with eth1
    name-description ( RW): 
              bridge ( RO): xenbr1
uuid ( RO)                : f10bb18a-d444-4a76-4aa7-8c36376c7d99
          name-label ( RW): Pool-wide network associated with eth0
    name-description ( RW): 
              bridge ( RO): xenbr0
uuid ( RO)                : e6ce997b-0011-798d-3aa1-6e8ea9c12762
          name-label ( RW): iSCSI A
    name-description ( RW):
              bridge ( RO): xenbr2
uuid ( RO)                : 409a8ee6-93ec-e858-5343-e80237156605
          name-label ( RW): Main Network (Bond)
    name-description ( RW):
              bridge ( RO): xapi1

You can see that the iSCSI (Bond) and Main Network (Bond) are the pif's that I am looking for:
iSCSI (Bond): aa0dec75-2f96-9de2-4517-ef21bedacc79
Main Network (Bond): 409a8ee6-93ec-e858-5343-e80237156605

# Find the physical interface (a bond is considered physical)
xe pif-list host-name-label=servername
uuid ( RO)                  : 6c9200d5-1830-d072-671a-e462691a8ff4
                device ( RO): bond0
    currently-attached ( RO): false
                  VLAN ( RO): 112
          network-uuid ( RO): 22d8b51f-ec0c-16a8-47c7-5a42093bef37
uuid ( RO)                  : da276895-3868-0f71-600e-29588bd90f81
                device ( RO): eth3
    currently-attached ( RO): false
                  VLAN ( RO): -1
          network-uuid ( RO): 0e7530ad-7506-00bb-5d69-8975e4ecaf32
uuid ( RO)                  : 752cd4b8-1ce1-71df-d95e-43690f88496d
                device ( RO): eth2
    currently-attached ( RO): false
                  VLAN ( RO): -1
          network-uuid ( RO): e6ce997b-0011-798d-3aa1-6e8ea9c12762
uuid ( RO)                  : 3ed85977-2ca1-336e-9283-c66f45393c23
                device ( RO): eth0
    currently-attached ( RO): false
                  VLAN ( RO): -1
          network-uuid ( RO): f10bb18a-d444-4a76-4aa7-8c36376c7d99
uuid ( RO)                  : b94e8e38-7ba1-7d0b-4c5a-9ada852afb3e
                device ( RO): bond1
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): aa0dec75-2f96-9de2-4517-ef21bedacc79
uuid ( RO)                  : 68dc47bb-bed0-8b4a-fbe9-d975c53849ee
                device ( RO): bond0
    currently-attached ( RO): false
                  VLAN ( RO): 12
          network-uuid ( RO): 237cdc19-3b84-0c97-7a98-842a97aa63e7
uuid ( RO)                  : 34cee247-2175-3bf6-99cb-f001ca124022
                device ( RO): bond0
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 409a8ee6-93ec-e858-5343-e80237156605
uuid ( RO)                  : 465f824a-43ab-9f58-cc15-5edaaaf9bcd4
                device ( RO): eth1
    currently-attached ( RO): false
                  VLAN ( RO): -1
          network-uuid ( RO): bf731568-91f5-1109-a6e8-f56d2b1a2ba9

Now find the networks in the pif-list:
bond0: 34cee247-2175-3bf6-99cb-f001ca124022
bond1: b94e8e38-7ba1-7d0b-4c5a-9ada852afb3e

Once you have the physical bond interfaces you need to add the parameter to use link-aggregation. The other-config needs to be in the order listed below because of how the xenserver's python script applies the configuration when setting up the interface. If it's done in the opposite order the /etc/sysconfig/network-scripts/ifcfg-bond0 will be all out of whack and the interface will not come up properly.

xe pif-param-set other-config:bond-use_carrier=1 other-config:bond-downdelay=200 other-config:bond-updelay=200 other-config:bond-miimon=100 other-config:bond-lacp_rate=1 other-config:bond-mode="802.3ad" uuid=34cee247-2175-3bf6-99cb-f001ca124022
xe pif-param-set other-config:bond-use_carrier=1 other-config:bond-downdelay=200 other-config:bond-updelay=200 other-config:bond-miimon=100 other-config:bond-lacp_rate=1 other-config:bond-mode="802.3ad" uuid=b94e8e38-7ba1-7d0b-4c5a-9ada852afb3e

There you have it. The way I add the bonds, currently until there is official support in xenserver is to add the management interface to the normal bond while the switch ports are still not configured for link-agg. As soon as the management interface is using the bond0 I then run the xe pif-param-set on the bonds, then I reboot the server. As the server is rebooted I set the ports to link-agg. Then the servers come up properly.