{"id":1183,"date":"2013-09-18T00:08:09","date_gmt":"2013-09-17T23:08:09","guid":{"rendered":"http:\/\/www.bussink.ch\/?p=1183"},"modified":"2018-05-23T17:57:19","modified_gmt":"2018-05-23T15:57:19","slug":"infiniband-in-the-lab","status":"publish","type":"post","link":"https:\/\/www.bussink.ch\/?p=1183","title":{"rendered":"InfiniBand in the lab…"},"content":{"rendered":"

Okay, the original title was going to be ‘InfiniBand in the lab… who can afford 10\/40 GbE’. I’ve looked in the past at 10GbE switches, and nearly pulled the trigger a few times. Even now the prices of the switches like the Netgear Prosafe or Cisco SG500X are going down, the cost of the 10GbE Adapter is still high. Having tested VSAN in the lab, I knew I wanted more speed for the replication and access to the data, than what I experienced.\u00a0 The kick in the butt in network acceleration that I have used is InfiniBand.<\/p>\n

If you search on ebay, you will find lots of very cheap InfiniBand host channel adapters (HCA) and cables. A Dual 20Gbps adapter will cost you between $40 and $80 dollars. The cables will vary between $15 and upto $150 depending of the type of cables. One of the interesting fact is that you can use InfiniBand in a point-to-point configuration. Each InfiniBand network needs a Subnet Manager, This is a configuration for the network, akin a Fabric Channel Zoning.<\/p>\n

InfiniBand Data Rates<\/strong><\/p>\n

An InfiniBand link is a serial link operating at one of five data rates: single data rate (SDR), double data rate (DDR), quad data rate (QDR), fourteen data rate (FDR), and enhanced data rate (EDR).<\/p>\n

    \n
  1. 10 Gbps or Single Data Rate (SDR)<\/li>\n
  2. 20 Gbps or Dual Data Rate (DDR)<\/li>\n
  3. 40 Gbps or Quad Data Rate (QDR)<\/li>\n
  4. 56 Gbps or Fourteen data rate (FDR)<\/li>\n
  5. 100Gbps or Enhanced data rate (EDR)<\/li>\n
  6. In 2014 will see the announcement of High data rate (HDR)<\/li>\n
  7. And the roadmap continues with next data rate (NDR)<\/li>\n<\/ol>\n

    There is a great entry InfiniBand on wikipedia<\/a> that discuss in larger terms the different signaling of InfiniBand.<\/p>\n

    InfiniBand Host Channel Adapters<\/strong><\/p>\n

    Two weeks ago, I found a great lead, information and that pushed me to purchased 6 InfiniBand adapters.<\/p>\n

    \n
    \n
    \"Item\u00a0 3x Mellanox InfiniBand MHGH28-XTC Dual Port DDR\/CX4 (PCIe Gen2) at $50.<\/div>\n<\/div>\n<\/div>\n
    \n
    \n
    \"Item 3x Mellanox InfiniBand MCX354A-FCBT CX354A Dual Port FDR\/QDR\u00a0 (PCIe Gen3) at $300.<\/div>\n<\/div>\n<\/div>\n

    InfiniBand Physical Interconnection<\/strong><\/p>\n

    Early InfiniBand used copper CX4 cable for SDR and DDR rates with 4x ports \u2014 also commonly used to connect SAS (Serial Attached SCSI) HBAs to external (SAS) disk arrays. With SAS, this is known as an SFF-8470 connector, and is referred to as an “InfiniBand-style” Connector.<\/p>\n

    \n
    \n
    \"Item Cisco 10GB CX4 to CX4 Infiniband Cable 1.5 m<\/div>\n<\/div>\n<\/div>\n

    The latest connectors used with up to QDR and FDR speeds 4x ports are QSFP (Quad SFP) and can be copper or fiber, depending on the length required.<\/p>\n

    InfiniBand Switch<\/strong><\/p>\n

    While you can create a triangle configuration with 3 hosts using Dual Port cards like Vladan Sege<\/a>t (@Vladan) writes in his very interesting article Homelab Storage Network Speed with InfiniBand<\/a> I wanted to see how a InfiniBand switch would work. I only invested in the following older Silverstorm 9024-CU24-ST2 that supports only 10Gbps SDR port. But it has 24x of them. Not bad for a $400 switch that supports 24x 10Gbps ports<\/span>.<\/p>\n

    \n
    \n
    \"Item<\/div>\n
    SilverStorm 10Gbps 24 port InfiniBand switch 9024-CU24-ST2<\/div>\n<\/div>\n<\/div>\n

    In my configuration each Dual Port Mellanox MHGH28-XTC (DDR Capable) will connect to my SilverStorm switch at only SDR 10Gbps speed, but I have two ports from each hosts. I can also increase the amount of hosts connected to the switch, and use a single Subnet Manager and single IPoIB (IP over InfiniBand) network addressing scheme. At the present time, I think this single IPoIB network addressing might be what is important for the implementation of VSAN in the lab.<\/p>\n

    Below you see the IB Port Statistics with three vSphere 5.1 hosts connected (1x cable per ESXi as I’m waiting on a 2nd batch of CX4 cables).<\/p>\n

    \"Silverstorm<\/a><\/p>\n

    The surprise I had when connecting to the SilverStorm 9024 switch is that it did not have the Subnet Manager. But thanks to Raphael Schitz<\/a> (@hypervisor_fr) who has successfully with the work & help of others (William Lam & Stjepan Gro\u0161<\/a>) and great tools (ESX Community Packaging Tool<\/a> by Andreas Peetz<\/a> @vFrontDE), repackaged the OpenFabrics Enterprise Distribution OpenSM<\/a> (Subnet Manager) so that it can be loaded on vSphere 5.0 and vSphere 5.1. This vSphere installable VIB can be found in his blog article\u00a0 InfiniBand@home votre homelab a 20Gbps<\/a> (In French).<\/p>\n

    The Link states in the screenshot above went to active, once the ib-Opensm<\/strong> was installed on the vSphere 5.1 hosts, the MTU<\/strong> was set and the partitions.conf<\/strong> configuration file written. Without Raphael’s ib-opensm, my InfiniBand switch would have been alone and not passed the IPoIB traffic in my lab.<\/p>\n

     <\/p>\n

    Installing the InfiniBand Adapters in vSphere 5.1<\/strong><\/p>\n

    Here is the process I used to install the InfiniBand drivers after adding the Host Channel Adapters. You will need three files. The first is the InfiniBand OFED Driver for VMware vSphere 5.x from Mellanox. The 2nd is the<\/p>\n

      \n
    1. VMware’s Mellanox 10Gb Ethernet driver supports products based on the Mellanox ConnectX Ethernet adapters<\/a><\/li>\n
    2. Mellanox InfiniBand OFED Driver for VMware vSphere 5.x<\/a><\/li>\n
    3. OpenFabrics.org Enterprise Distribution’s OpenSM for VMware vSphere 5.1<\/a> packaged by Raphael Schitz<\/a><\/li>\n<\/ol>\n

      You will need to transfer these three packages to each vSphere 5.x host, and install them using the esxcli command line. Before installing the VMware Mellanox ConnectX dcrive, you need to unzip the file, as it’s the offline zip file you want to supply in the ‘esxcli software vib’ install command. I push all the files via SSH in the \/tmp folder. I recommend that the host be put in maintenance mode, as you will need to reboot after the drivers are installed.<\/p>\n

      \"esxcli<\/a><\/p>\n

      The commands are<\/p>\n