ESXi Multi-NIC & Multi-VLAN vMotion on UCS

I’ve been deploying a Cisco UCS Chassis with multiple Cisco B230 M2 Blades. Yet the uplinks switches of the Fabric Interconnect are medium-Enterprise sized Switches, and not some Nexus 5K or better. In a vSphere 5.0 cluster designs you add one or more NICs to the vMotion interface. With the enhancements of Sphere 5.0 you can combine multiple 1G or 10G network cards for vMotion, and get better performance.

Duncan Epping wrote on the 14th December 2011 on his site
[quote]”I had a question last week about multi NIC vMotion. The question was if multi NIC vMotion was a multi initiator / multi target solution. Meaning that, if available, on both the source and the destination multiple NICs are used for the vMotion / migration of a VM. Yes it is!”[/quote]

I was a bit worried by having my ESXi 5.0 vMotion traffic go up the Fabric Interconnect from my source Blade, across the network switches and back down the Fabric Interconnect and the target Blade. I decided to create two vmkernel port for vMotion per ESXi, and segregate them in two VLAN. Each VLAN is only used inside one Fabric Interconnect.

vNIC Interface eth4 for vMotion-A on Fabric A (VLAN 70)

vNIC Interface eth4-vMotionA

vNIC Interface eth5 for vMotion-B on Fabric B (VLAN 71)

vNIC Interface eth5-vMotionB

And now let’s try this nice configuration.

The VM that would be used for testing purposes is a fat nested vESX with 32 vCPU and 64GB of memory (named esx21). It is vMotion’ed from esx12 (Source network stats in Red) towards esx11 (Target network stats in Blue).

The screenshot speaks for itself… we see that the vMotion uses both NICs and VLANs to transfer the memory to esx11. It flies at a total speed of 7504MbTX/s to 7369MbRX/s in two streams. One stream cannot pass the 5400Mb/s rate, because of the limitation of the Cisco 2104XP FEX and the 6120XP Fabric Interconnect. Each 10Gb link is used by two B230 M2 blades.

If you want to learn how to setup Multi-NIC vMotion, check out Duncan’s post on the topic.

Thanks go to Duncan Epping (@duncanyb) and Dave Alexander (@ucs_dave) for their help.

  • Paul

    Always great to see someone crunch the numbers. Well done and thank you for confirmation!

  • Jturver

    Hi, not sure about you comment “One stream cannot pass the 5400Mb/s rate, because of the limitation of the Cisco 2104XP FEX and the 6120XP Fabric Interconnect” – I seem to be able to get up to 8.5Gb with a single vMotion stream, single vMotion vmkernel NIC, single VLAN, and across fabrics with the same equipment? I there soem documentation on this limitation?
    Many thanks for the post

    • In my case, I have the 4 Links going from the Cisco 2104XP to the Fabric Interconnect 6120XP, but I have 8 blades in the 5108 Chassis. Meaning that I have two Blades that have to share the 10G link to Fabric A.
      Because I only had a stack of Cisco 3750X above the Fabric Interconnect, I didn’t have the bandwidth to allow the VLAN for vMotion on Fabric A to cross to a VLAN on Fabric B.

  • Werner

    To achieve the same you could have used the same VLAN, but configure vmnic4/vmnic5 active/passive on vSwitch3 and vmnic5/vmnic4 active/passive on vSwitch4, right? This would give failover of both vSwitches to fabric interconnect #2 in case of failure of fabric interconnect #1. What would be pros and cons of that alternative?

    • I don’t like to use Active/Passive configs with ESXi. The ESXi kernel is more than capable to use Active/Active.

      If one Fabric Interconnect reboots/dies it does not matter if at that time your vMotion traffic only uses a single vNIC. and having the two active vMotion NICs on the same Fabric Interconnect will not make it go faster (unless you have a uber UCS config with two 2208 FEX with all 8 server ports configured in a port channel.)

      As said in a reply above… I don’t see anything wrong with the using the same VLAN on both Fabric
      Interconnect, but you need to make sure you use two different C classe
      networks that don’t overlap, and or do not route the VLAN on the upstream switches.

  • Eric

    Good info! Can you describe why multi NIC vMotion is leaving the FI if configured with 1 VLAN in UCS? Thank you!

    • I have one question for you, if you have Multi vMotion NIC, do you use contigious IP addresses, or are you using two different IP range with non-overlapping network masks ?

      If you contigious IP addresses ie esx01 at 192.168.10.1 and 192.168.10.2 and esx02 at 192.168.10.3 and 192.168.10.4, there is no way that you can garantee that vMotion traffic from esx01 on .1 is going to esx02 on .2 and not .3. and that traffic needs to flow over the upstream network switches from FabricInterconnect-1 to the FabricInterconnect-2

      I don’t see anything wrong with the using the same VLAN on both Fabric Interconnect, but you need to make sure you use two different C classe networks that don’t overlap.