In this post, I will describe the storage design I use for my virtual infrastructure lab. I have been using the Community Edition of NexentaStor for the past two and a half year already. And I can tell already in the first paragraph of this post, that it’s a very impressive storage solution, which can scale to your needs and based on the budget you are ready to allocate to it.
I have played with various virtual storage appliances (VSA) from NetApp and EMC, and I used Openfiler 2.3 (x86-64) prior to moving to NexentaStor in my lab over 2.5 years ago. I was not getting the storage performance from the VSA, and it was difficult to add disks and storage to these VSA. The Community Edition of NexentaStor supports 18 TB of usable storage without requiring a paying license (you do need to register your Community Edition with Nexenta to get a license). I don’t believe a lot of people are hitting this limit in their labs. In addition, since NexentaStor 3.1, the VAAI primitives are supported with iSCSI traffic. There simply is no other way to test VAAI in a virtualization lab without spending some serious money.
Here are the current release notes for NexentaStor 3.1.2 and you can download the NexentaStor Community Edition 3.1.2 to give it a go. Version 4 of NexentaStor is planned for the summer of 2012. It will use the Illumos. I’m looking forward to the next release of NexentaStor.
Here is a screenshot of my storage layout I’m now using. I used to have a single large RAIDZ2 configuration last year, which was giving me a lot of space, but I found the system lacking on the write side, so I exported all my virtual machines from the NexentaStor server, and reformatted the storage using mirror of disks to improve the write speed.
As you can see in the previous screenshot, my tank zvol is composed of four set of mirrored disks, and I’m using two Intel SSD 520 60GB. One as the L2ARC cache, and one as the ZLOG cache disk.
Because I don’t have more than 16GB of RAM in the HP ProLiant ML150 G5, I decided not to use the De-dupe functionality of the NexentaStor. From Constantin’s blog, it seems like a good rule of thumb is 5GB of dedupe in ARC/L2ARC per TB of deduped pool space. I have about 3.6TB of disk, so that’s would require about 18GB (if I want to keep the De-dupe tables in RAM instead of the L2ARC).
I have enough CPU resources with the Quad-Core Xeon 5410 (@ 2.33GHz) to run the compression on the storage.
NexentaStor and VMware’s vStorage API for Array Integration (VAAI)
Nexenta introduced support for VMware’s vStorage API for Array Integration (VAAI). Specifically four SCSI commands have been added to improve the performance of several VMware operations when using iSCSI or FC connections. Following is a brief summary of the four functions, as described in the NexentaStor 3.1 release notes.
I’m covering the setting up of the iSCSI on the NexentaStor for vSphere in a separate post: Configuring iSCSI on Nexenta for vSphere 5.
vSphere 5 iSCSI Configuration
Here are some screenshots of how I setup the iSCSI configuration on my vSphere 5 cluster. The first one is the iSCSI Initiator with two iSCSI network cards.
Then I presented four 601GB LUNs to my vSphere 5 Cluster. You can see in the following screenshot those four LUNs with LUN ID 1,2,3,4, while the small 4GB LUNs with ID 7,8,9 are the RDM LUNs I’m presenting for the I/O Analyzer tests. We see that the Hardware Acceleration is Supported.
For each of these 601GB presented LUN, I modified the Path Selection from Most Recently Used (VMware) to Round Robin (VMware) and we see that all four paths are now Active (I/O) paths. iSCSI traffic is load balanced on both iSCSI network interfaces.
To see the load balancing across both iSCSI Network cards, I can quickly demo it with the IO Analyzer 1.1 doing a Max Throughput test (Max_Throughput.icf). I will show the results later in the post, but let’s first have a peak at the ESXTOP running from a vMA against my ESXi host. On this host, the two iSCSI vmkernel nics are vmk3 and vmk4.
and with the vSphere 5 Client, we can see that the traffic is using both the physical vmnic4 and vmnic5.
VMware’s Fling I/O Analyzer 1.1 benchmarking
These are the test I ran on my infrastructure with VMware’s Fling I/O Analyzer 1.1. The Fling I/O Analyzer deploys a virtual appliance running Linux, with a WINE implementation of the iometer on it. I highly recommend that you watch the following I/O Analyzer vBrownBag by Gabriel Chapman (@bacon_is_king) from March 2012 to understand how you can test your infrastructure.
Here are the three screenshots of the IO Analyzer running MAX_IOPS.icf (512b block 0% Random – 100% Read) against my NexentaStor. While it gives nice stats, and I’m the proud owner of a 55015 IOPS storage array, it’s not representative of day to day workload that the NexentaStor gives me.
In the next graphic, the 2nd set of tests to the right is the Max IOPS Test. There is a spike. But we clearly see that when pushing the system with 512 Byte Sequential Read, the Throughput is down from the Max Throughput test I used earlier.
And now lets the results in the IO Analyzer for the Max_IOPS.icf test. Result is 55015 Read IOPS.
Maximum Write Throughput
This test will measure the maximum Write Throughput to the NexentaStor server Using the Max Write Throughput test (512K, 100% Sequential, 100% Write)
This is the screenshot from the vMA showing the load balancing Write traffic.
vSphere 5 Client Performance chart of the iSCSI Load Balancing (Network Chart)
vSphere 5 Client Performance chart of the ESXi Write Rate (Disk Chart)
Here is the view from the Nexenta General Status, where you have two speedometers. Notice that the CPU is running at 42% due to Compression being enabled on my ZVOL.
And the result from the IO Accelerator for the Max Write Throughput Test. 75 MB/s or 151 WriteIOPS.
SQL Server Load
Now le’ts look at the SQL Server 64K test run on the I/O Analyzer. The test uses 64K blocks
Here is the vSphere 5 Client performance chart for the Disk. We see a nice 66% througput on Read and 33% on Write.
and the results giving us a nice 1354 total IOPS (893 ReadIOPS and 461 WriteIOPS).
Max IOPS from Three ESXi Hosts
One final test with IO Analyzer is to run three concurrent tests across three ESXi host on the same NexentaStor server. I’m using the Max_IOPS test (512b block 0% Random – 100% Read). We will notice that running the same test from Three sources instead of a single ESXi host, will result in a lower total IOPS result. So instead of a 55015 IOPS, we are getting and average of 12800 IOPS per host or a total of 38400 IOPS. Not bad at all for a lab storage server.
and the results are
I installed the Bonnie++ benchmark took directly on the Nexenta server, and I ran it multiple times, with a 48GB data file (so it’s three times larger than the RAM of my server).
Here are my results, and I apologize already for the very wide formatting this next HTML table will do to this post.
|Version 1.96||Sequential Output||Sequential Input||Random
|Sequential Create||Random Create|
|Size||Per Char||Block||Rewrite||Per Char||Block||Num Files||Create||Read||Delete||Create||Read||Delete|
|K/sec||% CPU||K/sec||% CPU||K/sec||% CPU||K/sec||% CPU||K/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||98||99||381672||85||262061||82||254||99||597024||69||6300||125||16||7013||42||+++++||+++||12665||60||12612||62||+++++||+++||14078||56|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||101||99||383428||86||265270||83||250||97||595142||69||7034||142||16||7742||46||+++++||+++||12861||60||13023||65||+++++||+++||13884||56|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||101||99||380309||85||256859||80||250||98||593337||69||10018||154||16||7290||44||+++++||+++||12650||61||12324||62||+++++||+++||14074||57|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||102||99||379416||85||255191||79||248||97||579578||68||5067||106||16||7103||43||+++++||+++||12781||61||13253||66||+++++||+++||14503||59|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||101||99||387046||86||257276||80||255||99||557928||65||11517||187||16||7070||42||+++++||+++||12651||61||13029||64||+++++||+++||13838||56|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||101||99||381530||85||258287||80||255||99||550162||64||10903||179||16||7314||45||+++++||+++||12640||60||11857||59||+++++||+++||13789||57|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||101||99||381556||86||258073||80||253||99||561755||65||5144||108||16||6805||41||+++++||+++||12625||60||12844||63||+++++||+++||14847||59|
|Nexenta2 HP ML150 G5 Tank&Compression&Cache&Log||48G||101||99||383434||85||259550||80||253||99||553803||65||5869||126||16||6724||40||+++++||+++||12236||59||11574||57||+++++||+++||13511||54|
The test was run on a ZVOL that had the Compression enabled, It was backed with a Intel SSD 520 60GB disk for the L2ARC Cache and a Intel SSD 520 60GB disk for the zlog.
There are some spikes here and there. but in average, the Bonnie++ is telling me that local storage access is capable of
In the past week, Chris Wahl over at http://wahlnetwork.com/ has writen four excellent articles about using NFS, Load Balancing and the Nexenta Community Edition server. I highly recommend you look up these articles to see how you can improve your Nexenta experience