Network core switch Cisco Nexus 3064PQ

Here is my new network core switch for the Home Datacenter, a Cisco Nexus 3064PQ-10GE.

Cisco Nexus 3064PQ-10GE (48x SFP+ & 4x QSFP+)

Cisco Nexus 3064PQ-10GE (48x SFP+ & 4x QSFP+)

But before I speak more about the Cisco Nexus 3064PQ-10GE, let me just bring you back in time… Two years ago, I purchased a Cisco SG500XG-8F8T 16-port 10-Gigabit Stackable Managed Switch. This was first described in my Homelab 2014 build. This was my most expensive networking investment I ever did. During the past two years, as the lab grew, I used the SG500XG and two SG500X-24 for my networking stack. This stack is still running on the 1.4.0.88 firmware.

sg500xg_stack

During these past two years, I have learned the hard way that network chipsets for 10GbE using RJ-45 cabling was outputting so much more heat than the SFP+ chipset. My initial Virtual SAN Hybrid implementation using a cluster of three ESXi host with Supermicro X9SRH-7TF (Network chipset is Intel X540-AT2) crashed more than once, when the network chipset became so hot that I lost my 10G connectivity, but the ESXi host kept on running. Only a powerdown & cool off of the motherboard, would allow my host to restart with the 10G connectivity. This also lead me to expand the VSAN Hybrid cluster from three to four hosts and to have a closer look at the heating issues when running 10G over RJ45.

Small business network switches with 10GBase-T connectivity are more expensive than the more enterprise oriented SFP+ switch, but they also output so much more heat (Measured in BTU/hr). Sure once the 10GBase-T switch is purchased, the cost of Category 6A cables is cheaper than getting the Passive Copper cables, who are limited to 7 meters.

The Cisco SG500XG-8F8T is a great switch as it allows me to connect using both RJ-45 and SFP+ cables.

As the lab expanded, I started to ensure that my new hosts have either no 10GBase-T adapters on the motherboard, or use the SFP+ adapter (Like my recent X10SDV-4C-7TP4F ESXi host). I have started using the Intel X710 Dual SFP+ adapters on some of my host. I like this Intel network adapter, as the network chipset gives out less heat than previous generations chipset, and has a firmware update function that can done from the command prompt inside of vSphere 6.0.

This brings me to the fact that I was starting to run out of SFP+ ports as the labs expands. I have found on ebay some older Cisco Nexus switch, and the one that caught my eye for it’s amount of ports, it price and it’s capabilities is the Cisco Nexus 3064PQ-10GE. These babies are going for about $1200-$1500 on ebay now.

3064pq_on_ebay

The switch comes with 48-ports SFP+ and 4-ports in QSFP+ format. These four ports can be configured in either 16x10G using fan-out cables or 4x40G. This is a software command that can be put on the switch to change from one mode to the other.

Here is my switch with the interface output. I’m using a Get-Console Airconsole to extend the console port to my iPad over Bluetooth.

nexus_3064pq_10g_40g-1

My vSphere 6.0 host is now connected to the switch using an Intel XL710-QDA2 40GbE network adapter and a QSFP+ copper cable.

esxi_40G

I’m going to use the four QSFP+ connectors on the Cisco Nexus 3064PQ-10GE to connect my Compute cluster with NSX and VSAN All-Flash.

3064_10g_40g_show_int

 

The switch came with NX-OS 5.0(3)U5(1f).

3068_nx-os

 

Concerning the heat output of the Cisco Nexus 3064PQ-10GE (datasheet) I was pleasantly surprised to note that it’s output is rather small at 488 BTU/hr when all 48 SFP+ are used. I also noted that the noise level of the fans was linked to the fan speed and the charge of the switch. Going from 59 dBA at 40% duty cycle to 66 dBA at 60% duty cycle to 71 dBA when at 100% duty cycle.

Here is the back of the Cisco Nexus 3064PQ-10GE. I did purchase the switch with a DC power (top of switch to the right), because the switch I wanted had both the LAN_BASE_SERVICES and the LAN_ENTERPRISE_SERVICES license. I sourced two N2200-PAC-400W-B power supply from another place.

nexus_3064pq_back-1

Link to the Cisco Nexus 3064PQ Architecture.

 

Speed testing 40G Ethernet in the Homelab

In my previous post, I described the building of two Linux virtual machines to benchmark the network. Here are the results.

homelab_network_1g_10g_40g_iperf_testing

 

The first blip, is running iperf to the maximum speed between the two Linux VMs at 1Gbps, on separate hosts using Intel I350-T2 adapters.

The second spike (or vmnic0), is running iperf to the maximum speed between two Linux VMs at 10Gbps. The two ESXi hosts are using Intel X540-T2 adapters.

The third mountain (or vmnic4) and most impressive result is running iperf between the Linux VMs using 40Gb Ethernet. The two ESXi hosts are using Mellanox ConnectX-3 VPI adapters.

The Homelab 2014 ESXi hosts, uses a Supermicro X9SRH-7TF come with an embedded Intel X540-T2. We can more closely see the  results of the iperf test at 10Gbps in the following picture.

homelab_network_10g_iperf_testing

I also got last summer from Ebay, a set of Mellanox ConnectX-3 VPI Dual Adapters for $300. These cards support InfiniBand 40Gb/s and 56Gb/s, and Ethernet at 10Gb/s and 40Gb/s. By default, vSphere 5.5 recognizes these adapters as 40Gb Ethernet adapters. And I really wanted to test these adapters at 40Gb Ethernet… and the results are great. I can push upto 37.3 Gbits/sec thru a single 40Gb Ethernet link, or 4299 MBytes/sec. Just have a peak at the following screenshot.

homelab_network_40g_iperf_testing

I guess having 40Gb Ethernet for vMotion is too fast…  The vMotion of a 12GB VM takes 15-16 seconds, of which only 3 seconds are used for the memory transfer, the rest is the memory snapshot, processes freeze, cpu register cloning and the rest.

All the test run at 10Gb Ethernet and 40Gb Ethernet where done with Jumbo Frames. For 40Gb Ethernet it makes real (x 2.5) difference in bandwidth.

This was a fun piece to lab in the homelab.

Upgrading Mellanox ConnectX firmware within ESXi

Last summer, while reading the ServeTheHome.com website, I saw a great link to Ebay for Mellanox ConnectX-3 VPI cards (MCX354A-FCBT). These cards where selling at $299 on ebay. I took three of the awesome cards. These Mellanox ConnectX-3 VPI adapters where simply too good to be true… Dual FDR 56Gb/s or 40/56GbE using PCIe Generation 3 slots. Having three of these Host Card Adapters without a InfiniBand switch is limiting.

With my new Homelab 2014 design, I now have two vSphere hosts that have PCIe Generation 3 slots, and using a simple QSFP+ Fiber Cable, I can create a direct point-to-point connection between the two vSphere hosts.

The Mellanox Firmware Tools (MFT) that can run within the vSphere 5.5 and allow to check the state of the InfiniBand adapter and even update the firmware.

MFT for vSphere

Installing the tools is very straight forward.

# esxcli software vib install -d /tmp/mlx-fw/MLNX-MFT-ESXi5.5-3.5.1.7.zip

Install Mellanox MST

Unfortunately it requires a reboot.

The next steps going to be to start the MST service, check the status of the of the Mellanox devices and query them to check the current level of firmware.

I don’t need to have the Mellanox MST driver running all the time, so I will simply start it using /opt/mellanox/bin/mst start.

Next we will query the state of all Mellanox devices in the host using /opt/mellanox/bin/mst status -v from which we will get the path to the devices.

We then use the flint tool to query the devices to get their stats.

/opt/mellanox/bin/flint -d /dev/mt40099_pci_cr0 hw query

and

/opt/mellanox/bin/flint -d /dev/mt40099_pci_cr0 query

which returns us the current Firmware version and the GUIDs and MACs for the host card adapters.

Mellanox firmware upgrade 01

Well as I’m running only FW Version 2.10.700 its time to upgrade this firmware to release 2.30.8000

 /opt/mellanox/bin/flint -d /dev/mt4099_pci_cr0 -i /tmp/mlx-fw/fw-ConnectX3-rel-2_30_8000-MCX354A-FCB_A1-FlexBoot-3.4.151_VPI.bin burn does the trick.

Mellanox firmware upgrade 02

And we can quickly check the new running firmware on the InfiniBand adapter.