General ZFS ARC Information

Here are two more screenshots that I just took on my infrastructure, while doing the initial backup of a 40GB virtual machine with VMware Data Recovery.

• I had the Nexenta management interface on the General Stats
• Running Richard Elling’s zilstat.ksh via SSH at the top left
• Running the arcstat.pl by Mike Harsch via SSH at the left
• And at the bottom left, you can glimpse the VMware Data Recovery job running on a 40GB VM (Initial run).

zilstats.ksh and arcstat.pl while running a VMware Data Recovery initial backup

I also took the opportunity to screenshot the disk and Virtual LUN stats page on the Nexenta page.

Disks and Virtual LUNs

These stats above are far from the most impressive results I’ve had with my NexentaStor server.

In this post, I will describe the storage design I use for my virtual infrastructure lab. I have been using the Community Edition of NexentaStor for the past two and a half year already. And I can tell already in the first paragraph of this post, that it’s a very impressive storage solution, which can scale to your needs and based on the budget you are ready to allocate to it.

I have played with various virtual storage appliances (VSA) from NetApp and EMC, and I used Openfiler 2.3 (x86-64) prior to moving to NexentaStor in my lab over 2.5 years ago. I was not getting the storage performance from the VSA, and it was difficult to add disks and storage to these VSA. The Community Edition of NexentaStor supports 18 TB of usable storage without requiring a paying license (you do need to register your Community Edition with Nexenta to get a license).  I don’t believe a lot of people are hitting this limit in their labs. In addition, since NexentaStor 3.1, the VAAI primitives are supported with iSCSI traffic. There simply is no other way to test VAAI in a virtualization lab without spending some serious money.

Here are the current release notes for NexentaStor 3.1.2 and you can download the NexentaStor Community Edition 3.1.2 to give it a go. Version 4 of NexentaStor is planned for the summer of 2012. It will use the Illumos. I’m looking forward to the next release of NexentaStor.

Hardware.

• My current implementation of NexentaStor is currently using an HP ProLiant ML150 G5 with a single quad-core Xeon 5410 (@2.33 GHz and 12MB of L2 Cache) and 16GB of ECC Memory.

• I’m using a LSI SAS 9201-16i HBA Card to connect my various disks in the system.

• My current hard disks are three year old 1TB SATA disks. They are definitely the weak point of my infrastructure at the present time, and I should really replace these aging disks by bigger and faster disks.

• I recently added three performant Intel 520 Series SSD. I took three 60GB disks. I’ve got Intel SSD in the past, and they are still very reliable, so the choice was not difficult. The 60GB versions of these disks are speced at 6700 IOPS RandomWrite, and 12000 IOPS RandomRead (I took the lowest numbers from the various Intel documentation). Getting larger disks would result in better IOPS performances and the drive would have more lifetime writes, but that would increase the cost of my storage.

• On the Network side, I have added an Intel based Dual gigabit server network card. My management traffic and NFS traffic arrive on the mainboard network card, and my iSCSI stack is presented using two IP addresses on the very good Supermicro AOC-SG-i2 network card. The Supermicro AOC-SG-i2 has dual Intel 82575EB chips. The iSCSI traffic is set to use a 9000 MTU, and I have an EtherChannel trunk (2x1Gbps) across the network switches (two Cisco SG300-28) from the HP ProLiant ML150 G5 to the ESXi servers in a second room.

Storage Layout.

Here is a screenshot of my storage layout I’m now using. I used to have a single large RAIDZ2 configuration last year, which was giving me a lot of space, but I found the system lacking on the write side, so I exported all my virtual machines from the NexentaStor server, and reformatted the storage using mirror of disks to improve the write speed.

zpool status & zpool list

As you can see in the previous screenshot, my tank zvol is composed of four set of mirrored disks, and I’m using two Intel SSD 520 60GB. One as the L2ARC cache, and one as the ZLOG cache disk.

Because I don’t have more than 16GB of RAM in the HP ProLiant ML150 G5, I decided not to use the De-dupe functionality of the NexentaStor. From Constantin’s blog, it seems like a good rule of thumb is 5GB of dedupe in ARC/L2ARC per TB of deduped pool space. I have about 3.6TB of disk, so that’s would require about  18GB (if I want to keep the De-dupe tables in RAM instead of the L2ARC).

I have enough CPU resources with the Quad-Core Xeon 5410 (@ 2.33GHz) to run the compression on the storage.

NexentaStor and VMware’s vStorage API for Array Integration (VAAI)

Nexenta introduced support for VMware’s vStorage API for Array Integration (VAAI). Specifically four SCSI commands have been added to improve the performance of several VMware operations when using iSCSI or FC connections. Following is a brief summary of the four functions, as described in the NexentaStor 3.1 release notes.

• SCSI Write Same: When creating a new virtual disk VMware must write’s zeros to every block location on the disk. This is done to ensure no residual data exists on the disk which could be read by the new VM. Without Write Same support the server’s CPU must write each individual block which consumes a lot of CPU cycles. With the Write Same command VMware can direct the storage array to perform this function, offloading it from the CPU and thereby saving CPU cycles for other operations. This is supported in ESX/ESXi 4.1 and later.
• SCSI ATS: Without ATS support, when VMware clones a VM it must lock the entire LUN to prevent changes to the VM while it is being cloned. Howerver, locking the entire LUN affects all other VM’s that are using the same LUN. With ATS support VMware is able to instruct the array to lock only the specific region on the LUN being cloned. This allows other operations affecting other parts of the LUN to continue unaffected. This is supported in ESX/ESXi 4.1 and later.
• SCSI Block Copy: Without Block Copy support, cloning a VM requires the server CPU to read and write each block of the VM consuming a lot of server CPU cycles. With Block Copy support VMware is able to instruct the array to perform a block copy of a region on the LUN corresponding to a VM. This offloads the task from the server’s CPU thereby saving CPU cycles for other operations. This is supported in ESX/ESXI 4.1 and later.
• SCSI Unmap: Provides the ability to return freed blocks in a zvol back to a pool. Previously the pool would only grow. This enables ESXi to destroy a VM and return the freed storage back to the zvol. ESXi 5.0 and later support this functionality.

I’m covering the setting up of the iSCSI on the NexentaStor for vSphere in a separate post: Configuring iSCSI on Nexenta for vSphere 5.

vSphere 5 iSCSI Configuration

Here are some screenshots of how I setup the iSCSI configuration on my vSphere 5 cluster. The first one is the iSCSI Initiator with two iSCSI network cards.

vSphere 5 iSCSI Initiator VMkernel Port Bindings

Then I presented four 601GB LUNs to my vSphere 5 Cluster. You can see in the following screenshot those four LUNs with LUN ID 1,2,3,4, while the small 4GB LUNs with ID 7,8,9 are the RDM LUNs I’m presenting for the I/O Analyzer tests. We see that the Hardware Acceleration is Supported.

For each of these 601GB presented LUN, I modified the Path Selection from Most Recently Used (VMware) to Round Robin (VMware) and we see that all four paths are now Active (I/O) paths. iSCSI traffic is load balanced on both iSCSI network interfaces.

vSphere 5 iSCSI LUNs Path Selection set to Round Robin (VMware)

To see the load balancing across both iSCSI Network cards, I can quickly demo it with the IO Analyzer 1.1 doing a Max Throughput test (Max_Throughput.icf). I will show the results later in the post, but let’s first have a peak at the ESXTOP running from a vMA against my ESXi host. On this host, the two iSCSI vmkernel nics are vmk3 and vmk4.

IO Analyzer doing Max Throughput test. Load balanced iSCSI Traffic with Nexenta CLI

and with the vSphere 5 Client, we can see that the traffic is using both the physical vmnic4 and vmnic5.

VMware’s Fling I/O Analyzer 1.1 benchmarking

These are the test I ran on my infrastructure with VMware’s Fling I/O Analyzer 1.1. The Fling I/O Analyzer deploys a virtual appliance running Linux, with a WINE implementation of the iometer on it. I highly recommend that you watch the following I/O Analyzer vBrownBag by Gabriel Chapman (@bacon_is_king) from March 2012 to understand how you can test your infrastructure.

Maximum IOPS

Here are the three screenshots of the IO Analyzer running MAX_IOPS.icf (512b block 0% Random – 100% Read) against my NexentaStor. While it gives nice stats, and I’m the proud owner of a 55015 IOPS storage array, it’s not representative of day to day workload that the NexentaStor gives me.

In the next graphic, the 2nd set of tests to the right is the Max IOPS Test. There is a spike. But we clearly see that when pushing the system with 512 Byte Sequential Read, the Throughput is down from the Max Throughput test I used earlier.

IO Analyzer Max_IOPS Test from the vSphere 5 Client

And now lets the results in the IO Analyzer for the Max_IOPS.icf test. Result is 55015 Read IOPS.

IO Analyzer results for Max_IOPS.icf

Maximum Write Throughput

This test will measure the maximum Write Throughput to the NexentaStor server Using the Max Write Throughput test (512K, 100% Sequential, 100% Write)

This is the screenshot from the vMA showing the load balancing Write traffic.

IO Analyzer running MaxWriteThroughPut from the vMA

vSphere 5 Client Performance chart of the iSCSI Load Balancing (Network Chart)

IO Analyzer running MaxWriteThroughPut (Network) from the vSphere 5 Client

vSphere 5 Client Performance chart of the ESXi Write Rate (Disk Chart)

IO Analyzer running MaxWriteThroughPut (Disk from the vSphere 5 Client

Here is the view from the Nexenta General Status, where you have two speedometers. Notice that the CPU is running at 42% due to Compression being enabled on my ZVOL.

Nexenta GUI Status when running the Max Write Throughput test

And the result from the IO Accelerator for the Max Write Throughput Test. 75 MB/s or 151 WriteIOPS.

IO Analyzer Results for Max Write Throughput with the Nexenta

SQL Server Load

Now le’ts look at the SQL Server 64K test run on the I/O Analyzer. The test uses 64K blocks

IO Analyzer running SQL Server 64K test on Nexenta

Here is the vSphere 5 Client performance chart for the Disk. We see a nice 66% througput on Read and 33% on Write.

IO Analyzer running SQL Server 64K test on Nexenta (Disk Chart)

and the results giving us a nice 1354 total IOPS (893 ReadIOPS and 461 WriteIOPS).

IO Analyzer SQL Server 64K Results

Max IOPS from Three ESXi Hosts

One final test with IO Analyzer is to run three concurrent tests across three ESXi host on the same NexentaStor server. I’m using the Max_IOPS test (512b block 0% Random – 100% Read). We will notice that running the same test from Three sources instead of a single ESXi host, will result in a lower total IOPS result. So instead of a 55015 IOPS, we are getting and average of 12800 IOPS per host or a total of 38400 IOPS. Not bad at all for a lab storage server.

3x IO Analyzer running on dedicated Host in parallel

and the results are

3x IO Analyzer running on dedicated Host in parallel Results

Bonnie++ Benchmarking

I installed the Bonnie++ benchmark took directly on the Nexenta server, and I ran it multiple times, with a 48GB data file (so it’s three times larger than the RAM of my server).

Here are my results, and I apologize already for the very wide formatting this next HTML table will do to this post.

The test was run on a ZVOL that had the Compression enabled, It was backed with a Intel SSD 520 60GB disk for the L2ARC Cache and a Intel SSD 520 60GB disk for the zlog.

There are some spikes here and there. but in average, the Bonnie++ is telling me that local storage access is capable of

• Sequential Block Reads: 381530K/sec (381MB/sec)
• Sequential Block Writes: 550162K/sec (550MB/sec)
• Rewrite: 255191K/sec (255MB/sec)
• Random Seeks: 5067/sec

Additional Resources:

In the past week, Chris Wahl over at http://wahlnetwork.com/ has writen four excellent articles about using NFS, Load Balancing and the Nexenta Community Edition server. I highly recommend you look up these articles to see how you can improve your Nexenta experience

1. Misconceptions on how NFS behaves on vSphere by Chris Wahl
2. Load balancing NFS deep dive in both a single subnet by Chris Wahl
3. Load balancing NFS deep dive with multiple subnet by Chris Wahl
4. and NFS on vSphere – Technical Deep Dive on Load Based Teaming by Chris Wahl

The virtual machine is running

• Windows Server 2008 R2 SP1
• vCenter 5.0 Update 1
• SQL Server 2008 R2 SP1 (10.50.2792)
• and the whole suite of vCenter services (vum, syslog, dump, web service).

The virtual machine was created  on an ESX 3.5 (Build 604481) and was configured as a VM Version 4.  The target platform was a new ESXi 5.0 Update 1 host (Build 623860). So we cold migrated the vCenter to the new system, via a shared VMFS3 datastore.

At this point, the virtual machine was running fine as a VM Version 4 on the ESXi 5.0 Update 1.

I then started the upgrade process, with the installation of the VMware Tools, to ensure I had all the proper drivers in the VM. I then powered off the virtual machine, and upgraded the hardware to VM Version 8.

vCenter - VM Version 8

The system restarted but there was an issue with the various services. I could not open the network settings, I could not uninstall the VMware Tools as the Windows Installer service was not running. My data and database log disks where not visible, I could not open the disk management control panel.

After much troubleshooting, restarting the virtual machine in safe mode and various other tests, my colleague found this very interesting article Windows Server 2008 computer hang during startup while “applying computer settings” and services configured to start automatically fail to start http://support.microsoft.com/kb/2004121 

The following two paragraphs are taken from the Microsoft Support Article.

Cause

The problems described in the symptoms section occur because of a lock on the Service Control Manager (SCM) database.  As a result of the lock, none of the services can access the SCM database to initialize their service start requests. To verify that a Windows computer is affected by the problem discussed in this article, run the following command from the command Prompt:

sc querylock

The output below would indicate that the SCM database is locked:

QueryServiceLockstatus – Success

IsLocked : True

LockOwner : .\NT Service Control Manager

LockDuration : 1090 (seconds since acquired)

Let me fix it myself

you can modify the behavior of HTTP.SYS to depend on another service being started first.  To do this, perform the following steps:

NOTE: Please ensure that you make a backup of the registry / affected keys before making any changes to your system.

After having made the registry modification and a final restart, the virtual machine was working again as expect. This was a very strange and bizarre error I have never heard someone run into. So here it is resumed, and may it be usefull someday to someone else…

But this morning, once I was out of range of my house, the WiFi dropped and the iPad3 tried to reach my telecom 3G network. My Carrier network (Swisscom) was showing up as 3G but I kept receiving the “Could not activate cellular data network“.

1) On my initial migration from the iPad1 I used the Reset Network settings described below.

The trick is to go to the Settings \ General \ Reset tab, and to select the Reset Network Settings. This will cause the iPad3 to reboot, but then the Carrier 3G network will work.

2) But I’m not going wipe my Network settings again and again. Only a Power-Off Power-On of the iPad currently fixes this issue. We all hope Apple will fix this with an iOS 5.1.1 release.

In the following screenshots we will use the yum update tool to make sure we upgrade to RHEL 5.7 only.

The first screenshot shows the current kernel 2.6.18-308.el5 for RHEL 5.6, and the configuration of the yum.conf file that has an explicit exclude=redhat-release-5Server* rule. We also see that we now have the redhat-release-5Server-5.6.0.3.

Current vCD-Cell settings for RHEL 5.6

We will now modify the /etc/yum.conf so that we can download the redhat-release-5Server-5.7.0.3.x86_64.rpm file. We comment out the exclude file, and we install immediately the release file for RHEL 5.7

vCD-Cell upgrading from RHEL 5.6 to RHEL 5.7

Now it’s important to immedialty renable the exclusion of the redhat-release-5Server, so that you will not by accident upgrade to RHEL 5.8

Ensure that yum cannot retrieve RHEL 5.8

Now you can run the yum upgrade to your own pace, and be sure that you are staying on the supported release of Red Hat Enterprise Linux for the vCloud Director 1.5.1

vCenter Operations Manager 5.0 vApp Start Order Settings

I’m adding this post, because in the past few days I have had to do a Storage vMotion of the Analytics VM, and I had to temporarily remove it from the vApp. Once I had migrated the Analytics VM, I insert the VM back in the vApp. But this changed the default start order, and the Analytics VM had default settings, such as Shutdown Action operation:PowerOff, and a different Startup sequence. You can see the default settings in the screenshot below, when I add another VM to the vApp.

vApp Start Order Settings you do NOT want

So make sure that your vApp Startup settings are properly configured when you tamper with the vApp.

I’ve create numerous virtual machines named DC1, as I’m trying to use the Microsoft Windows Server “8” Beta Base Configuration Test Lab Guide (TLG) that is located at http://go.microsoft.com/fwlink/p/?LinkId=236358.

I have used these VMs with the VMware Tools from the TechPreview, without the VMware Tools, and with a Custom implementation without the SVGA graphic drivers. I’ve attemped my test on both the Windows Server 8 Beta with GUI and in Server Core.

Workstation 2012 Tech Preview and Windows Server 8 Beta AD-DS install blank screen bug

As soon as I try to install and configure the Active Director Domain Services, the VM needs to reboot. Once it has rebooted, it goes in a blank screen, and there is nothing I can do. Workstation thinks the VM is running, but there is no response via the GUI in the VM, no response to ping traffic to the VM or RDP.

I installed Workstation Technology Preview 2012 on two different computers and re-downloaded the en_windows_server_8_beta_x64_dvd_810648.iso from Microsoft twice. I just can proceed with using the Workstation Technology Preview 2012 to test Windows Server 8 Domain controllers.

I made a small video of the process, which is appended to this article.

Windows Server 8 Beta (Server Core) AD-DS install inside Workstation 2012 Tech Preview

In addition it’s available on Youtube at http://www.youtube.com/watch?v=6qvptvC0Usc

Here I’m trying to install the Active Directory Domain Services on a Windows Server 8 Beta running inside the VMware Workstation 2012 Tech Preview. The install of the AD-DS and DNS service work fine, but when the domain controller reboots, there is no GUI left. In this VM the VMware Tools where not installed.

Update: Well in VMware Workstation 2012 Tech Preview, if you select the a Windows 7 version or Windows Server 2008 version instead of the Windows 8 setting, your V will NOT go black screen on the dcpromo.

To ensure that your vCloud Director 1.5 stays on the Red Hat Enterprise Linux 5.6 Release and only download patches for the operating system, we need to add a single line to the /etc/yum.conf file.

Disable RHEL 5.6 Release Upgrade

I simply add the following line in /etc/yum.conf

exclude=redhat-release-5Server*

This will exclude all newer Red Hat Releases from getting installed by yum & the Red Hat Network.

I hope this will save you so unneeded trouble.

I have seem my share of transaction log databases for VMware vCenter go haywire, such that the Roll-Up jobs are not running anymore (Check your History Log) and the transaction log databases explodes. My personal worse situation was last year at a client that didn’t check their database and the transaction log database run out of storage on the disk when it passed the 90GB. There are procedures on the VMware Knowledge Base on how to compact and roll-up these huge transaction database, but it takes a lot of time. In most cases, we cut out losses and just purge the transactions logs.

Coming back to my Maintenance plan. We will create to sets of database maintenance plans, one for the System Dababases and one for the User databases. I need to thank my friend Eric Krejci for showing me how to separate the two maintenance plans.

System Maintenance Plan

We need to connect to our database server using the SQL Server Management Studio program. And from the Management folder, select the Maintenance Plan and start the Wizard.

Start Maintenance Plan Wizard

The System databases is comprised of the Master, Model, MSDB and TempDB databases. These database don’t change much, but I will select to make a Twice Weekly maintenance and Backup Plan. Please note that the MSDB database contains all the Stored Procedures for your vCenter & Update Manager database.

Define Maintenace Plan

And let’s Schedule the Plan for two runs per week on Tuesday evening and Friday evening.

Job Schedule Properties

You can select any other pattern that you wish.I for one also use VMware Data Recovery 2.0 for making daily backup of my virtual machines, so I make sure that my  VMware Data Recovery schedule does not run on my databases between 23:00 and 01:00.

Now we can select the various Maintenance Tasks we want to run.

Select Maintenance Tasks

I have selected

• Check Database Integrity
• Shrink Database
• Update Statistics
• Clean Up History
• Back Up Database (Full)
• Maintenance Cleanup Task.

And I have changed their Order around on Select Maintenance Task Order step.

Select Maintenance Task Order

So we run

1. Check Database Integrity
2. Update Statistics
3. Back Up Database (Full)
4. Shrink Database
5. Maintenance Cleanup Task
6. Clean Up History

Now let’s configure the Maintenance Tasks – Define Check Database Integrity. I have selected for this first Maintenance Plan the System Databases.

Define Database Check Integrity Task - System Databases

We now Define Update Statistics Task for the System Databases

Define Update Statistics Task - System Databases

The next step is the definition of the back up job. Define Back Up Database (Full) Task. Please note that we have added the option to create a sub-directory for each database, and to verify the backup integrity. I have also modified the Backup File Extention to BAK_FULL_SYS so that we can make better use and more flexible backup cleanup maintenance job later in this article.

Define Back Up Database (Full) Task - System Databases

There is always a good discussion if you have enough Compute power to create a compressed backup or not.

Now that we have a good full backup for the system databases we can do some database shrinkage. Define Shrink Database Task.

Define Shrink Database Task - System Databases

Now remember that we modified the Backup File Extension earlier. We we will now Define Maintenance Cleanup Task to erase all System Databases backups that are older than two weeks, and we will use the various sub-folders for the backups.

Define Maintenance Cleanup Task - System Databases

And  last we Define History Cleanup Task for the whole SQL Server 2008 R2 instance. I did not modify the settings of this tab. This Maintenance Task will cleanup the Backup and Restory History, the SQL Server Agent job history and the Maintenance Plan History.

Define History Cleanup Task

We will also save a copy the Maintenance Plan actions to a text file in the same directory as where the backup files are stored.

Select Report Option for Maintenance Plan

We now have a resume of the Maintenance Plan we can complete.

Maintenance Plan Wizard Complete

We see the new job in the Maintenance Plans section and the new job in the SQL Server Agent

Maintenance Plans & SQL Agent Jobs

User Maintenace Plan

We now attack the User Databases Maintenance Plan. We start our Maintenance Plan Wizard and start the definition of the plan properties. I’m creating a Maintenance Plan for the Users Databases that will create a Differential Back Up every day, and a Full Back Up on Friday.

Users Database Maintenance Plan Properties

I modify the Schedule so that the main part of this Maintenance Plan including the Full Back Up happens each Friday. I will then later add a subplan to do the Differential plan each day.

User Databases Maintenance Plan - Job Schedule

We now add the various Maintenance Tasks for our Users Databases.

Select Maintenance Tasks

I have selected

• Check Database Integrity
• Shrink Database
• Rebuild Index
• Update Statistics
• Back Up Database (Full)
• Maintenance Cleanup Task

And we Select Maintenance Task Order to move down the Shrink Database task after the Back Up Database (Full).

Select Maintenance Task Order

So we run

1. Check Database Integrity
2. Rebuild Index
3. Update Statistics
4. Back Up Database (Full)
5. Shrink Database
6. Maintenance Cleanup Task

The first Task to run is the Database Check Integrity Task where we select the Users Databases

Database Check Integrity Task - User Databases

We then Rebuild Index Task for the Users Databases

Rebuild Index Task - User Databases

We Define Update Statistics Task for the User Databases.

Update Statistics Task - User Databases

We now do the Back Up Database (Full) Task for the User Databases. Note that we use sub-directories for each database, we changed teh Backup File extionsion to BAK_FULL_USR and we verify the integrity of the backup.

Back Up Database (Full) Task - User Databases

Once we have the Full Back Up of the User Databases we can launch the Shrink Database Task.

Shrink Database Task - User Databases

We now setup the Maintenance Cleanup Task for the User Databases so that we keep only the last two weekly full backups.

Maintenance Cleanup Task - User Databases

And we save the Maintenance Plan Report to the job_history directory.

Maintenance Plan Report Path

We now have a complete Maintenace Plan ready.

User Databases Maintenance Plan Wizard Complete

This creates the new Maintenance Plan and the SQL Agent Job.

Maintenance Plan & SQL Agent Jobs

We now select to Modify the User Databases – MaintenancePlan

Modify User Databases Maintenance Plan

And let’s quickly rename the Subplan_1 to Subplan_Weekly in the Subplan menu.

Rename Subplan_1 to Subplan_Weekly

So we can now Add Subplan to this Maintenance Task

Add Subplan_Daily

And we edit the Job Schedule to run everyday but Friday at the same time.

Job Schedule Subplan_Daily

We will now drag and drop the Back Up Database Task into the Subplan_Daily

Back Up Database Task in Subplan_Daily

We now edit the Back Up Database Task

Edit Back Up Database Task

And we modify the Back Up Database Task for Differential Jobs, we also make sure the backups are written in their correct directories, that they are verified, and that the Backup File Extension is BAK_DIFF_USR.

Back Up Database Task - User Databases - Differential Job

We now add the Maintenance Cleanup Task to this Subplan_Daily job and Linked it to the Back Up Database Task.

Add Maintenance Cleanup Task

And we will edit the Maintenance Cleanup Task so that we erase the old BAK_DIFF_USR files.

Maintenance Cleanup Task 1 - Backup Files

We add a 2nd Maintenance Cleanup Task to clean up the old text reports that are older than 4 weeks.

Maintenance Cleanup Task 2 - Text Reports

We are now done with the User Databases Maintenance Plan. Do NOT forget to SAVE the Maintenace Plan before quiting it.

We now have two specific SQL Server Agent Jobs.

SQL Server Agent Jobs

We will now run the Maintenace Plan Jobs. We start with the System Database job using Start Job at Step…

Running Maintenace Plan Jobs - System Databases

And for the User Databases we will first start the Full Back Up Task, before doing the Differential Back Up Task.

Running Maintenance Plan - User Database - Subplan_Weekly

Running Maintenance Plan - User Database - Subplan_Daily

When we check the Backup folder we now have a full back of the System Databases and User Databases (Full and Differential).

vCenter Server Backup Full and Diff

There you are with a Maintenance Plan for the SQL Server 2008 R2 running your vCenter, Update Manager and vCloud Director databases.

I hope this will help you.

I have to thank once more my friend Eric Krejci as we have discussed this topic extensively a few months ago and he already wrote the same article on vCenter and SQL Backup and Maintenance on his web blog.

I recently followed Julian Wood’s excellent series on how to sign certificates for vCenter and Update Manager. Generating the SSL Certificates for vCenter Operations Manager goes along the same lines, but there are changed and maybe some configuration changes on the vCOPS UI-VM.

Julian recommends to install the latest 64-bit version of the OpenSSL Windows Binaries. Retrieve the Win64 OpenSSL v1.0.1 Light for Windows tool on the vCenter with it’s per-requisite Visual C++ 2008 Redistributables (x64) from Microsoft.com

Once the OpenSSL v1.0.1 Light is installed, we can add an System Environment Variable, so that the OpenSSL tool can find the path to the OpenSSL configuration file. Because I’m going to use the OpenSSL tool on the vCenter to generate the SSL Certificates for various VMware appliance, I need the variable to stay permanent. From the Control Panel on the vCenter, I add a new System Environment Variable like follows.

Adding the OPENSSL_CONF environment variable in the Control Panel

So that the next time you start the Command Prompt to generate OpenSSL Certificates, the variable is already present.

Checking OPENSSL_CONF variable

One of the best information I learned from Julian’s document is the modification of the openssl.cfg to add the option to use two subjectAltName for the DNS resolution. This allows the user to get a valid certificate when you connect to the vCenter Operations Manager 5.0, using the Fully Qualified Domain Name or simply the short name of the server.

To use this feature you will need to edit the C:\OpenSSL-Win64\bin\openssl.cfg and add “req_extensions = v3_req” to the “[ req ]” section, and add “subjectAltName = DNS:vcops.vsphere.bussink.local,DNS:vcops” to the “[ v3_req]” section. I need to add that I also modify the default key length in the certificate request to 2048 bits.

Update (29/03/2012): I added to my subjectAltName, the iPAddress of my vCenter Operations Manager UI. You will get the information from the vCenter Managed Object Reference portal ExtensionManager value (See screenshot at the bottom of the post). The entry is of format DNS:192.168.1.18

Update (02/04/2012): Thanks to Josh Perkins excellent article “vCenter Operations Manager 5 vCenter Plugin uses IP instead of DNS hostname“. I have removed the IP address subjectAltName in the certificate request in the code above.

To create the Certificate file I used the following commands. Go to the bin directory of the OpenSSL tools. Generate a new Certificate Request while keeping the Cert Private key on your vCenter server. I’m generating the vCOPS private key with the 2048bit RSA algorithms and the SHA256 Message Digest algorithms.

Generate vCOPS Certificate Request

Once we have the Certificate Request for the vCenter Operations Manager, we can submit it to the Public Key Infrastructure for certification. There are two ways to it, once from the command prompt and via the Web interface of the PKI.

Command Prompt Certificate Request

Windows Server 2008 R2 has a simple tool, to submit the Certificate Request directly the Microsoft Root CA (Enterprise Mode).

On my Certificate Authority I have cloned the default WebServer Certificate Template, and named it OpenSSL. I have also modified it’s Validity Period, Renewal Period. See completely at the bottom of this post to get an explanation and description of these changes.

My Microsoft Certificate Authority implementation is configured so that Certificate Requests need to be authorized, so the Submit/Retrieve process is composed of two commands here: certreq -submit and certreq -retrieve, if your Certificate Authority is not setup with validation, the submission/retrieval process is done in a single command.

Submitting vCOPS Certificate Request from Command Prompt

At this point the Certificate has been submitted to the Root CA authority in the domain. Please note the RequestId number when you submit the Certificate Request. Once the Certificate has been authorized and generated you can retrieve it back to the vCenter.

Retrieve vCOPS Certificate from Command Prompt

If we open the vcops.cer in Windows, we can see that the Certificate has also proper Certificates in the Certification Path. This is important to ensure that browsers can validate the vCOPS Certificate all the way up to the Certificate Authority (with the Issuing CA is it’s an Intermediate Certification Authority).

Verify your vCOPS Certificate for the Certification Path

We now need to build a PKCS#12 container file with the Certificate, the Private Key and output it to the .PFX file.

Build vCOPS PKCS12 Container

vCenter Operations Manager 5.0 does not use the PKCS#12 file format, but the PEM format, and requires that the Private Key is not protect by password. So we re-transform the the .PFX with the Private Key into the .PEM format.

Transform vCOPS from PKCS12 Container to PEM format

At this point open the Administrator interface of vCenter Operations Manager on the SSL pane, and import the PEM certificate.

The url is https://vcops.<your-domain>/admin/

Importing SSL Certificate in vCOPS

But here comes a tricky part. It’s debugging time.

It is very possible that your Import of the OpenSSL Certificate fails with a General error occured. Like below.

OpenSSL Import General Error Occurred

What I found is that the apache2 Web Server on vCOPS did not like loading my SSL Certificate, because it saw that the certificate was for a FQDN that it could not figure out. I modified the /etc/hosts file to ensure apache2 got the proper hostname while starting up and therefore accepted the OpenSSL Certificates.

Modify /etc/hosts file on vCOPS

In the next screenshot you see the error messages from the apache2 at startup when it cannot figure out it’s name and when it does.

vCOPS apache2 startup with default /etc/hosts and modified /etc/hosts

You can always check the vCOPS log files at /var/log/vmware/ for issues.

In the screnshot below we see that I tried to install onces the vcops.pfx format, and then the vcops.pem certificate (@23:38:15), I then restarted the vCOPS Web Service and all is good after 23:46:13.

Checking the vcops-admin.log for SSL install issues

We can now connect to vCenter Operations Manager using the FQDN or the short-name.

Valid SSL Certificate for vCOPS

I have also found that once the OpenSSL Certificate has been changed, that the vCOPS Interface  feels much more reactive.

 

Appendix 1) – My OpenSSL Certificate Template

On my Active Directory Certificate Services I have cloned the default WebServer Certificate Template, and named it OpenSSL. I have also modified it’s Validity Period, Renewal Period and the need for the Certificate Authority Manager to approve all Certificate Requests.I highly recommend that you set the Validity Period for your Certificate Template. The CA Manager Approval really depends on your environment. As I sometimes do Auto-Enrollment tests for devices, I don’t want to pollute my Root CA with hundreds of superseding certificates.

OpenSSL Certificate Template Properties - Validity Period

OpenSSL Certificate Template Properties – CA Manager Approval

 

 

Appendix 2) – Retrieve the Root & Intermediate Certificate Authority Public Key using CertUtil

In this second appendix, I will briefly show how to retrieve the Root Certificate Authority Public Key from the command prompt. You should also retrieve the Intermediate CA if you have one.

certutil -ca.cert -config “domctrl01.vsphere.bussink.local\Bussink Root CA” RootCA.cer

Retrieve Certificate Authority Public Key RootCA.cer

 

Update on 16/03/2012. Changed the Win64 OpenSSL v1.0.1 Light tools.

Update 27/03/2012. Added a additional subjectAltName to the Certificate request. But my had my parameters wrong.

Update (27/03/2012): I have added a new subjectAltName on the to my openssl.cfg. I added the FQDN name of my vCenter server in the Certificate request. With vCenter Operations Manager 5.0, you get the integration within the vCenter Client in the Solutions & Applications section. The SSL Certificates will therefore be checked by the vCenter Client against the vCenter FQDN name.

Update 29/03/2012. Thanks for Kinsei for having raises the question on the topic of the SSL Certificate usage via the vCenter Client. The vCenter Operations Manager is connected to the vCenter Server not by an FQDN name, but by an IP Address. You can find the value when you connect to your vCenter server’s Managed Object Reference (mob) settings portal.

https://vcenter/mob/ Content ExtensionManager ExtensionList com.vmware.vcops

 Update (02/04/2012). Here is another update. Josh Perkins has written up a great article on how to ensure that your vCenter uses a FQDN or shortname to speak to your vCenter Operations Manager. This means that administrators and user on the vSphere Client do not get invalid SSL Certificate requests anymore. Thanks Josh !!