2013 Homelab refresh

Preamble

It’s now 2013, and it’s time to have a peak at my homelab refresh for this year.

 

Background

In the past three years, I’ve ran a very light homelab with VMware ESXi. I mainly used my workstation (Supermicro X8DTH-6F) with Dual Xeon 5520 @2.26Ghz (8-Cores) and 72GB of RAM to run most of the virtual machines and for testing within VMware Workstation, and only ran domain controllers and 1 proxy VM on a small ESXi machine, a Shuttle XG41. This gives a lot of flexibilty to run nearly all the virtual machines on a large beefed up workstation. There are quiet a few posts on this topic on various vExpert websites (I highly recommend Eric Sloof’s Super-Workstation).

I sometimes do play games (I’m married with a gamer), and when I do I have to ensure my virtual machines are powered down within VMware Workstation, as my system could and has crashed during games. Having corrupted VM is no fun.

 

Requirements

What I want for 2013 in the homelab, is a flexible environment composed of a few quiet ESXi hosts with my larger workstation being able to add new loads or test specific VM configuration. For this I need a infrastructure that is small, quiet and stable. Here are the requirements for my 2013 homelab infrastructure

  1. Wife Acceptance Factor (WAF)
  2. Small
  3. Quiet
  4. Power Efficient

Having purchased a flat, I don’t have a technical room (nothing like my 2006 computer room) or a basement. So having a few ESXi hosts on 24 hours a day, requires a high Wife Acceptance Factor. The system have to be small & quiet. In addition, if they are power efficient, it will make the utility bill easier.

 

Shuttle XH61V

The Shuttle XH61V is small black desktop based on the Intel H61 chipset. It comes in a 3.5L metal case with very quiet ventilators. You just need to purchase the Shuttle XH61V, a Intel 1155 Socket 65W processor, two memory SODIMMs (laptop memory) and local storage. Assembly can be done in less than 30 minutes.

Shuttle XH61V

Shuttle XH61V

The Shuttle XH61V comes with two NICs and support for a mSATA (Bootable) connector, a PCIe x1 slot, and two 2.5″ Devices. The Shuttle XH61V comes with two gigabit network cards. They are Realtek 8168 cards. These work flawlessly, but they do not support Jumbo frames.

Shuttle XH61V Back

Shuttle XH61V Back

For storage, I decided to boot from a mSATA device, and to keep a Intel SSD for a fast upper-tier local storage, and one large Hybrid 2.5″ Harddisk for main storage. I do have a Synology DS1010+ on the network that is the centralized NFS storage, but I want some fast local storage for specific virtual machines. It’s still early 2013, so I have not yet upgraded my older Synology or created a new powerful & quiet Nexenta Community Edition home server. On the next image you can see that three Shuttle XH61V take less space than a Synology DS1010+

Three Shuttle HX61V with Synology DS1010+

VMware ESXi installation

Installing VMware ESXi is done quickly as all the devices drivers are on the ESXi 5.1 VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso install cdrom.

ESXi 5.1 on XH61V

ESXi 5.1 on XH61V

Here is the Hardware Status for the Shuttle XH61V

ESXi XH61V Hardware Status

Here is an updated screenshot of my vSphere 5.1 homelab cluster.

Management Cluster

 

Bill of Materials (BOM)

Here is my updated bill of materials (BOM) for my ESXi nodes.

  • Shuttle XH61V
  • Intel Core  i7-3770S CPU @3.1Ghz
  • Two Kingston 8GB DDR3 SO-DIMM KVR1333D3S9/8G
  • Kingston 16GB USB 3.0 Key to boot ESXi (Change BIOS as you cannot boot a USB key in USB3 mode)
  • Local Storage Intel SSD 525 120GB
  • Local Storage Intel SSD 520 240GB
  • Local Storage Seagate Momentus XT 750GB

Planned upgrade: I hope to get new Intel SSD 525 mSATA boot devices to replace the older Kingston SSDnow when they become available.

 

Performance & Efficiency

In my bill of materials, I selected the most powerful Intel Core i7 processor that I could fit in the Shuttle XH61V. Because I’m running virtual appliances and virtual machines like vCenter Operations Manager, SQL Databases, Splunk. There are some less expensive Core i3 (3M Cache), Core i5 (6M Cache) or Core i7 (8M Cache) processor that would work great.

What is impressive, is that the Shuttle XH61V comes with a 90W power adapter. We are far from the 300W mini-boxes/XPC or even the HP MicroServer with their 150W power adapters. Only the Intel NUC comes lower with a 65W power adapter and a single gigabit network (@AlexGalbraith has a great series of post on running ESXi on his Intel NUC ).

Just for info, the Intel Core i7-3770S has a cpubenchmark.net score of 9312. Which is really good for a small box that uses 90W.

The Shuttle XH61V is also very quiet... it’s barely a few decibels above the noise of a very quiet room. To tell you the thru… the WAF is really working, as my wife is now sleeping with two running XH61V at less than 2 meters away. And she does not notice them… 🙂

 

Pricing

The pricing for a Shuttle XH61V with 16GB memory and a USB boot device (16GB Kingston USB 3.0) can be kept to a about $350 on newegg. What will increase the price is the performance of the LGA 1155 Socket 65W processor ( Core i3-2130 from $130 to Core  i7-3770S at $300) and what additional local storage you want to put in.

vSphere 5.1 Cluster XH61V

The sizing of the homelab in early 2013 is so far from the end of 2006 when I moved out of my first flat, when I had a dedicated Computer room.

Update 18/03/2012. DirectPath I/O Configuration for Shuttle XH61v BIOS 1.04

XH61v DirectPath I/O Configuration

XH61v DirectPath I/O Configuration

 

Update 22/03/2013.  mSATA SSD Upgrade

I’ve decided to replace the Intel 525 30GB mSATA SSD that is used for booting ESXi and to store the Host Cache with a larger Intel 525 120GB mSATA SSD. This device will give me more space to store the Host Cache and will be used as a small Tier for the Temp scratch disk of my SQL virtual machine.

The ‘published’ performance for the Intel 525 120GB mSATA are

Capacity
Sequential
Read/Write (up to)
Random 4KB
Read/Write (up to)
Form Factor
30 GB SATA 6 Gb/s       500 MB/s / 275 MB/s  5,000 IOPS / 80,000 IOPS mSATA
60 GB SATA 6 Gb/s       550 MB/s / 475 MB/s 15,000 IOPS / 80,000 IOPS mSATA
120 GB SATA 6 Gb/s       550 MB/s / 500 MB/s 25,000 IOPS / 80,000 IOPS mSATA
180 GB SATA 6 Gb/s       550 MB/s / 520 MB/s 50,000 IOPS / 80,000 IOPS mSATA
240 GB SATA 6 Gb/s       550 MB/s / 520 MB/s 50,000 IOPS / 80,000 IOPS mSATA
 Show More Detailed Product Specifications >

 

  • nice boxes!!

    • Not bad at all for a quiet & performant mini box.

    • Not bad at all for a quiet & performant mini box.I’ve not also decided to get some larger Intel 525 mSATA, so I can have more speed and faster latency for the Host Cache and other vmdk files on it.

  • Mark

    I’m considering getting one of these to run ESXi with an i5 3470T. Does the XH61V support VT-d? I’m not sure if VT-d is required, but at some point I want to possibly play with MythTV with a USB TV tuner so would hope it could be passed through to the VM.

    • Mark, it seems to be a great question.

      The Intel H61 Chipset which is the chipset on the XH61v does not support Intel Intel Virtualization Technology for Directed I/O (VT-d) as it’s explained on their website. Intel H61 Chipset http://ark.intel.com/products/52806/Intel-BD82H61-PCH

      The Intel H67 does not support VT-d, but the Intel Q67 does.

      Now looking at the DirecPath I/O Configuration on my ESXi 5.1.0 Build 1021289 server, I can configure the Passthrough for the Intel C200 chipset and Realtek 8168 Gigabit Ethernet interface. I have not tried it, and I currently do not have a screen attacked to the ESXi to reboot and check the BIOS right now. Let me attach a screenshot in this blog entry for you.

  • Pingback: Welcome to vSphere-land! » Home Lab Links()

    • Thanks a lot Eric.

  • Newer versions of the shuttle will come with Realtek 8111E Chipset which supports Jumbo Frames! 😉
    http://us.shuttle.com/barebone/Models/XH61V.html

    What power consumption do you have per shuttle barebone?

    Regards,
    Fred

    • The 3rd HX61v I got, had a newer V2 motherboard, but it did not come with Realtek 8111E chipset. So still stuck at 1500 MTU right now.

      I’ll plug a Kill-A-Watt meter to see how much the XH61v consumes soon.

      • Sorry Manfred for the late answer. I’ve put a power-meter and the shuttles with light CPU load (as always with Virtualization) consume from 37W to 40W. I haven’t seen anything above that load yet.

  • bergie1

    Love your setup!
    Does your system support vMotion? Do you know if there are specific requirements for the CPU to support it? Currently considering two Shuttle DS61 (smaller in size, almost same specs) and a cheaper CPU. Gonna use it with iSCSI on my Synology DS1812+ .

    • Yes, I’ve for vMotion support and FT support.

      The CPU needs to be a 65W processor, there is a list of supported CPU on the shuttle.com website. I personnaly use the i7-3770s, but they might be a bit overkill, maybe the 65W I5 might be a cheaper solution.

      I also use iSCSI with an older DS1010+ (as seen on the screenshots above).

      While the shuttle DS61 is smaller than the XH61v, it lacks a single 2.5″ drive.

      I currently use a 16GB USB3 (but connected in USB2 to boot) drive, a 120GB mSATA SSD, a 2.5″ 240GB SSD drive and a 2.5″ HDD drive. That means two full SSD devices and a single HDD.

      Trust me, the HX61v is just the ‘right’ size for the future… do not go the smaller DS61. You might regret it…

      • bergie1

        Thanks! I’m going to reconsider the DS61.

        So there are no other requirements other than the 65w limit for the Shuttle barebone? Since I’m not going to run more than a maximum of 5-6 vm’s (mostly linux) divided betweed the two hosts, i was considering the cheaper I3-2120T (35w) or even the Celeron G1610 (55w). Do you think those would work?

        • The supported CPU are on this list… I would stick to I5 if you can’t afford the I7 65w processors. This is because the I5 still has 4 cores, but 6M of cache. The I3 will drop this cache back to 3M, which will cause a perf impact on VMs.

          Here is the Intel ARK Comparaison on a I3, two I5 and a I7.

          http://ark.intel.com/compare/65512,65518,66168,65524

        • Yes it will work. But having only 2 pCPU for 5-6 VMs might be a bit light. It really depends on the workload you will have within your Linux VMs.

  • Graham

    How is the networking setup? Do you run all services (vMotion, IP storage, FT etc..) over 2 NICs?

    • Yes, I have currently a vSwitch with both active NICs, and i run vMotion, FT and iSCSI on them.

    • I have even toyed with the Delock MiniPCIe I/O PCIe full size 2x Gigabit LAN extension. Giving this little XH61v box, four network cards with the same chipset.

      The Delock MiniPCIe I/O PCIE full size 2x Gigabit LAN, uses the mSATA connector, and you have to keep the top of the XH61v open, but it works !!

      http://www.delock.de/produkte/F_929_Ethernet_95237/merkmale.html

      I personally prefer using the mSATA for a SSD device. But it’s an option.

      • nathan

        Just a note I have the same setup and grabbed myself a Delock MiniPCIe I/O PCIE full size 2x Gigabit LAN and I have it running the 2 network cables out the front of the pc via the empty cd bay. this means pc is still all sealed up (stackable), I did however remove the upper front flap for convenience (note it can be left on).

        of course this means two network leads run out the front of my shuttle but this is a small price for 4 nics

        https://www.dropbox.com/sh/h0a2ce47baw144f/KloB6zcCVN

        EDIT: totally forget to thanks for sharing this, the setup is awesome. Great website. also I injected the Realtek driver into 5.5 iso and all four nics work!!!

        • Thanks, I had not tought about running the Delock cable out front… more Delock MiniPCIe on the way now 🙂 Thanks a lot for the great idea.

          • nbajam

            Erik, I may have missed the post but could you share your network topology of these hosts, switch and nas??

            It would be great to see how you have it all setup, as I am still in the process. any advice would be appreciated

            Nathan

          • Behind this cluster of XH61v, I have a SG500-28 (Fanless) switch. For the NAS, you could use a Synology.
            I’m actually not using the Synology DS1010+ on the picture for VMs, as it’s a bit slow for VMs.
            I have build a Nexenta Community Edition storage server.

  • Chris’s Blog

    Hi Eric, just wanted to say thanks for a great post and a great find. Recently bought one of these boxes as my wife was unhappy with the noise of my existing lab servers (a pair of ML110’s). These Shuttle boxes make excellent lab servers. I went for the Intel mSATA and a Seagate Hybrid along with a regular SSD. Couldn’t be happier..well I could but I will have to wait for the second one to arrive for that 🙂

    • Awesome news. And maybe starting thinking about getting a USB key to boot the OS from. You might want to re-use that mSATA SSD for something cool.

  • Pingback: Lab update – Shuttle XH61V |()

  • Kevin

    I want to duplicate your Home Lab. Can I use any i5 processor on the Shuttle CPU Support List? Will they all support FT?
    -Thanks

    • I would stick to the Core i5-3550S or i5-3570S or i5-3470s to ensure you have the proper Intel Virtualization Technologies in the CPU. Check the Intel ARK archive for the CPU and make sure you support the VT-x technology.

      – Intel® Virtualization Technology (VT-x) Yes
      – Intel® Virtualization Technology for Directed I/O (VT-d) Yes
      – Intel® VT-x with Extended Page Tables (EPT) Yes

    • Yes you can use the i5 processors, chech the support list at Shuttle.com then head to Intel ARK to see specs of the processors.

  • Kevin

    Please give me your opinion on the Intel Core i3-3225 Ivy Bridge 3.3GHz LGA 1155
    55W Dual-Core Desktop. I know that I drop 2 cores and 5 MB cache but how do you think it would function for my esxi 5 boxes? Thanks.

    • I’m sure it’s going to work, but I would rather have 4 cores like an i5, because you can load about 5-7 nice VMs on a single XH61v. You will have CPU contention then.

    • It really depends on the work load you will put in your VMs.

  • Brian

    I just grabbed two xh61v with i5 procs. I really want more nics. Is there any option other than the delock?

    • I have not found another supplier in Europe that does a mSATA to Two-NIC.

    • Alex Galbraith in the UK is testing a jetway adapter, based on an intel chipset, but he was experiencing some stability issues. Will ask him about it tomorrow.

  • Romain

    Hi Erik,

    First of all, happy new year and all the best for 2015!

    Following a power outage, my Shuttle is not powering up anymore. I can here the disk spinning. I removed RAM and was expecting to get BIPS but nothing happened.
    I am afraid the motherboard died. Do you know which model of motherboard I could put in as a replacement? I could not find the details on Shuttle website.
    Thanks in advance!

    Cheers
    Romain

  • Pingback: Home Lab - vMBaggum()