A new beginning… joining VMware

After nearly 13 years at LANexpert, a Value Added Reseller and Integrator in the french speaking area of Switzerland, I’ll be starting a new chapter of my professional career. I’ll be joining VMware as a Solutions Architect on the 1st of October.

Over the better part of the last decade, I have played my part in building the Virtualization practice for LANexpert, into one of the first VMware Premier Partner in Switzerland. During this past decade, I have seen and pushed the changes in IT infrastructure from standalone server, to nearly full virtualized datacenter (not the network yet). Now I have the great opportunity to join the ‘mothership’ and keep pushing a technology that I trust and truly believe in.

I’m excited with the new challenge and the opportunity to meet a lot more new people as driven as me by virtualization, and sad to leave a great ‘family’ as LANexpert behind me. I also want to thanks all the people around me in the community, that have helped me directly or indirectly to grow and extend my technical expertise.

Erik Bussink

 

 

Creating a Linux Net benchmark VM

In this post, I will quickly explain, how I created my Virtual Machine under Linux, that I have and will use to benchmark some aspects of my new 2014 Homelab. First I download from the CentOS website, the latest version of the CentOS 6.5 64bit Net Install .ISO. This will allow me to install the Virtual Machine quickly with the packages I need.

The next step is to create a two Linux 64bit VMs on my vCenter. I selected to create a VMX-09 virtual machine, so that I can edit the network properties from the vCenter 5.5 Windows Client or the vSphere Web client. I create a two vCPU machine, because the application that I will be running for my network benchmarks is iperf, and is a single-threaded process, so the 2nd vCPU will be consumed by the operating system of the VM.

For Network Adapters, I select two VMXNET3 adapters, the first one will be used for management and baselining my perfs on a 1Gbps Ethernet, the 2nd one can be moved around from vSwitch to dVSwitch and from VMNIC to VMNIC. Note that I rather give two virtual sockets with one core, than one virtual socket with two cores. This will give you about 6% more performance for the VM.

vm_64bit_linux_01

Another small change I always do, is to optimize the Virtual Machine Monitor for the VMs. The VMM is a thin layer for each VM that leverages the the scheduling, memory management and the network stack in the VMkernel. So I change in the Options tab, the CPU/MMU Virtualization settings to force the use of Intel VT-x/AMD-V for instruction set virtualization and Intel EPT/AMD RVI for MMU virtualization. This will ensure that the VM gets the best optimized hardware supportfor the CPU and MMU. This should only be done on recent processors, when you are sure that your CPU/MMU supports EPT and VT-X. If that is not the case, then leave this setting to Automatic.

vm_64bit_linux_02_cpu-mmu

 

If you want to know more about these settings and many others, I highly recommend you read the great “vSphere High Performance Cookbook” by Prasenjit Sarkar (@stretchcloud) at Packt Publishing.

I just need to say that in the past few years, all my VMs and Templates get this setting by default on all my systems and my customer clusters.

Next, we need to boot the Linux machine with the CentOS Net Installer. I’m not going to explain all the steps needed for every Linux settings, just a few points. When you get the option to select the installation method we select the URL option.

CentOS Installation Method

It will then ask you to select the network card and will fetch an IP address from the network via DHCP before asking you to enter the URL. We will use the following URL

http://mirror.centos.org/centos/6.5/os/x86_64/

Enter URL

Once the install GUI has started make sure not to forget to put the 2nd Ethernet interface where you will be doing your iperf testing to a 9000 MTU. Otherwise your network performance results will be skewed.

nic_eth1_mtu
For my performance testing VMs, I let the OS select the default file partition scheme, this is not a VM requiring special sizing.

default_partition_scheme

I select the Desktop installation config for these test platforms.

desktop installation

Once you have finished installing the virtual machine, install the latest VMware Tools on it, before modifying the grub menu. I add the key work VGA=0x317 to all my linux machines kernel settings in grub.conf or menu.lst (OpenSuSE), so that the VM boots think it has a 1024×768 monitor. Even if I stay in the Console mode of Linux, it gives me more screen estate.

When you have Linux machines that run on 1Gbps ethernet, the default settings in the Linux kernel are fine, but if you want to optimize the network traffic for Linux for 10Gbps, there are a few System variables that we can fine tune. Lets edit the /etc/sysctl.conf and add six fields:

# Minimum, initial and max TCP Receive buffer size in Bytes

net.ipv4.tcp_rmem = 4096 87380 134217728

# Minimum, initial and max buffer space allocated

net.ipv4.tcp_wmem = 4096 65536 134217728

# TCP Moderage Receive Buffer Auto-Tuning

net.ipv4.tcp_moderate_rcvbuf=1

# Maximum Receive socket buffer size (size of BDP)

net.core.rmem_max = 134217728

# Maximum Send socket buffer size (size of BDP)

net.core.wmem_max = 134217728

# Maximum number of packets queueed on the input side

net.core.netdev_max_backlog = 300000

I’m going to use iperf to test the links between two machines, so for this set of machines, I disable the IPtables as I have multiple ports being used between the two linux test platforms. chkconfig iptables off will do the trick. A quick reboot and all the modifications will take effect.

Also as we will test the 10G Ethernet performance, both virtual machines are on a Distributed vSwitch (dVS), and the PortGroup is configured with a MTU set at 9000 (Jumbo Frames).

And before finishing this blog, I also make sure to use DRS Rules, so that the Linux VM 01 should runs on my ESX01 server, and the Linux VM 02 should run on my ESX02 server. Using the Should rule, allows me to quickly put a host in maintenance mode, while ensure that my performance virtual machines stay where they should.

To use the iperf (a very single threaded program) between two test hosts, start iperf on the first one as a service iperf -s , and on the second one, we use the commands iperf -m -i t300 -c IP_of_other_VM or iperf -m -i t300 -c IP_of_other_VM -fM to have the same results but in Bytes instead of bits.

Here is preliminary results using a 10G Ethernet interface between the two hosts (both hosts have an Intel X540-T2 adaper).

10g_results

 

 

Upgrading vCloud Director Cell from RHEL 5.6 to RHEL 5.7

With the release of vCloud Director 1.5.1 last night, the operating system for the vCloud Director Cell now supports Red Hat Enterprise Linux 5.7 (x86_64). If you are running your current cell with Red Hat Enterprise Linux 5.6, and you want to upgrade to the most recent release that is supported, here are the steps. Yet, you have to be careful not to upgrade to Red Hat Enterprise Linux 5.8, which as been release the 21st February 2012. RHEL 5.8 is not on the official supported list by VMware.

In the following screenshots we will use the yum update tool to make sure we upgrade to RHEL 5.7 only.

The first screenshot shows the current kernel 2.6.18-308.el5 for RHEL 5.6, and the configuration of the yum.conf file that has an explicit exclude=redhat-release-5Server* rule. We also see that we now have the redhat-release-5Server-5.6.0.3.

Current vCD-Cell settings for RHEL 5.6

We will now modify the /etc/yum.conf so that we can download the redhat-release-5Server-5.7.0.3.x86_64.rpm file. We comment out the exclude file, and we install immediately the release file for RHEL 5.7

vCD-Cell upgrading from RHEL 5.6 to RHEL 5.7

Now it’s important to immedialty renable the exclusion of the redhat-release-5Server, so that you will not by accident upgrade to RHEL 5.8

Ensure that yum cannot retrieve RHEL 5.8

Now you can run the yum upgrade to your own pace, and be sure that you are staying on the supported release of Red Hat Enterprise Linux for the vCloud Director 1.5.1

 

vCenter Operations Manager 5 vApp Start Order settings

When you deploy the vCenter Operations Manager 5.0 vApp in a vSphere 5 Cluster, the vApp import creates a few settings. Here is the screenshot of the default start order.

vCenter Operations Manager 5.0 vApp Start Order Settings

I’m adding this post, because in the past few days I have had to do a Storage vMotion of the Analytics VM, and I had to temporarily remove it from the vApp. Once I had migrated the Analytics VM, I insert the VM back in the vApp. But this changed the default start order, and the Analytics VM had default settings, such as Shutdown Action operation:PowerOff, and a different Startup sequence. You can see the default settings in the screenshot below, when I add another VM to the vApp.

vApp Start Order Settings you do NOT want

So make sure that your vApp Startup settings are properly configured when you tamper with the vApp.

 

Generating SSL Certificates for vCenter Operations Manager 5.0

Generating SSL Certificates for usage with vCenter, Update Manager and the ESXi host is one of those tasks that keeps being push away. Accepting the self-signed certificates is fine in most situation, but getting validated certificates means a whole lot of pop-ups disappear and surprise surprise, I have also found that the vCenter Operations Manager feels smother and faster.

I recently followed Julian Wood’s excellent series on how to sign certificates for vCenter and Update Manager. Generating the SSL Certificates for vCenter Operations Manager goes along the same lines, but there are changed and maybe some configuration changes on the vCOPS UI-VM.

Julian recommends to install the latest 64-bit version of the OpenSSL Windows Binaries. Retrieve the Win64 OpenSSL v1.0.1 Light for Windows tool on the vCenter with it’s per-requisite Visual C++ 2008 Redistributables (x64) from Microsoft.com

Once the OpenSSL v1.0.1 Light is installed, we can add an System Environment Variable, so that the OpenSSL tool can find the path to the OpenSSL configuration file. Because I’m going to use the OpenSSL tool on the vCenter to generate the SSL Certificates for various VMware appliance, I need the variable to stay permanent. From the Control Panel on the vCenter, I add a new System Environment Variable like follows.

Adding the OPENSSL_CONF environment variable in the Control Panel

So that the next time you start the Command Prompt to generate OpenSSL Certificates, the variable is already present.

Checking OPENSSL_CONF variable

One of the best information I learned from Julian’s document is the modification of the openssl.cfg to add the option to use two subjectAltName for the DNS resolution. This allows the user to get a valid certificate when you connect to the vCenter Operations Manager 5.0, using the Fully Qualified Domain Name or simply the short name of the server.

To use this feature you will need to edit the C:\OpenSSL-Win64\bin\openssl.cfg and add “req_extensions = v3_req” to the “[ req ]” section, and add “subjectAltName = DNS:vcops.vsphere.bussink.local,DNS:vcops” to the “[ v3_req]” section. I need to add that I also modify the default key length in the certificate request to 2048 bits.

[box] [ req ]

default_bits        = 2048

req_extensions = v3_req

[ v3_req ]

subjectAltName = DNS:vcops.vsphere.bussink.local, DNS:vcops, DNS:192.168.1.18

subjectAltName = DNS:vcops.vsphere.bussink.local, DNS:vcops

[/box]

Update (29/03/2012): I added to my subjectAltName, the iPAddress of my vCenter Operations Manager UI. You will get the information from the vCenter Managed Object Reference portal ExtensionManager value (See screenshot at the bottom of the post). The entry is of format DNS:192.168.1.18

Update (02/04/2012): Thanks to Josh Perkins excellent article “vCenter Operations Manager 5 vCenter Plugin uses IP instead of DNS hostname“. I have removed the IP address subjectAltName in the certificate request in the code above.

To create the Certificate file I used the following commands. Go to the bin directory of the OpenSSL tools. Generate a new Certificate Request while keeping the Cert Private key on your vCenter server. I’m generating the vCOPS private key with the 2048bit RSA algorithms and the SHA256 Message Digest algorithms.

[box] cd C:\OpenSSL-Win64\bin

openssl req -new -nodes -newkey rsa:2048 -sha256 -out vcops.csr -keyout vcops.key

[/box]

Generate vCOPS Certificate Request

Once we have the Certificate Request for the vCenter Operations Manager, we can submit it to the Public Key Infrastructure for certification. There are two ways to it, once from the command prompt and via the Web interface of the PKI.

Command Prompt Certificate Request

Windows Server 2008 R2 has a simple tool, to submit the Certificate Request directly the Microsoft Root CA (Enterprise Mode).

On my Certificate Authority I have cloned the default WebServer Certificate Template, and named it OpenSSL. I have also modified it’s Validity Period, Renewal Period. See completely at the bottom of this post to get an explanation and description of these changes.

My Microsoft Certificate Authority implementation is configured so that Certificate Requests need to be authorized, so the Submit/Retrieve process is composed of two commands here: certreq -submit and certreq -retrieve, if your Certificate Authority is not setup with validation, the submission/retrieval process is done in a single command.

[box]

certreq -submit -attrib “CertificateTemplate:WebServer” vcops.csr

or

certreq -submit attrib “CertificateTemplate:OpenSSL” vcops.csr[/box]

 

Submitting vCOPS Certificate Request from Command Prompt

At this point the Certificate has been submitted to the Root CA authority in the domain. Please note the RequestId number when you submit the Certificate Request. Once the Certificate has been authorized and generated you can retrieve it back to the vCenter.

[box]certreq -retrieve 16 vcops.cer [/box]

Retrieve vCOPS Certificate from Command Prompt

If we open the vcops.cer in Windows, we can see that the Certificate has also proper Certificates in the Certification Path. This is important to ensure that browsers can validate the vCOPS Certificate all the way up to the Certificate Authority (with the Issuing CA is it’s an Intermediate Certification Authority).

Verify your vCOPS Certificate for the Certification Path

We now need to build a PKCS#12 container file with the Certificate, the Private Key and output it to the .PFX file.

[box] openssl pkcs12 -export -in vcops.cer -inkey vcops.key -name vcops -out vcops.pfx[/box]

Build vCOPS PKCS12 Container

vCenter Operations Manager 5.0 does not use the PKCS#12 file format, but the PEM format, and requires that the Private Key is not protect by password. So we re-transform the the .PFX with the Private Key into the .PEM format.

[box] openssl pkcs12 -in vcops.pfx -inkey vcops.key -out vcops.pem -nodes[/box]

Transform vCOPS from PKCS12 Container to PEM format

At this point open the Administrator interface of vCenter Operations Manager on the SSL pane, and import the PEM certificate.

The url is https://vcops.<your-domain>/admin/

Importing SSL Certificate in vCOPS

 

But here comes a tricky part. It’s debugging time.

It is very possible that your Import of the OpenSSL Certificate fails with a General error occured. Like below.

OpenSSL Import General Error Occurred

What I found is that the apache2 Web Server on vCOPS did not like loading my SSL Certificate, because it saw that the certificate was for a FQDN that it could not figure out. I modified the /etc/hosts file to ensure apache2 got the proper hostname while starting up and therefore accepted the OpenSSL Certificates.

Modify /etc/hosts file on vCOPS

In the next screenshot you see the error messages from the apache2 at startup when it cannot figure out it’s name and when it does.

[box]/sbin/service apache2 restart [/box]

vCOPS apache2 startup with default /etc/hosts and modified /etc/hosts

 

You can always check the vCOPS log files at /var/log/vmware/ for issues.

In the screnshot below we see that I tried to install onces the vcops.pfx format, and then the vcops.pem certificate (@23:38:15), I then restarted the vCOPS Web Service and all is good after 23:46:13.

[box] tail /var/log/vmware/vcops-admin.log[/box]

Checking the vcops-admin.log for SSL install issues

We can now connect to vCenter Operations Manager using the FQDN or the short-name.

Valid SSL Certificate for vCOPS

I have also found that once the OpenSSL Certificate has been changed, that the vCOPS Interface  feels much more reactive.

 

Appendix 1) – My OpenSSL Certificate Template

On my Active Directory Certificate Services I have cloned the default WebServer Certificate Template, and named it OpenSSL. I have also modified it’s Validity Period, Renewal Period and the need for the Certificate Authority Manager to approve all Certificate Requests.I highly recommend that you set the Validity Period for your Certificate Template. The CA Manager Approval really depends on your environment. As I sometimes do Auto-Enrollment tests for devices, I don’t want to pollute my Root CA with hundreds of superseding certificates.

OpenSSL Certificate Template Properties - Validity Period

OpenSSL Certificate Template Properties – CA Manager Approval

 

 

Appendix 2) – Retrieve the Root & Intermediate Certificate Authority Public Key using CertUtil

In this second appendix, I will briefly show how to retrieve the Root Certificate Authority Public Key from the command prompt. You should also retrieve the Intermediate CA if you have one.

[box] certutil -ca.cert -config “domctrl01.vsphere.bussink.local\Bussink Root CA” RootCA.cer[/box]

Retrieve Certificate Authority Public Key RootCA.cer

 

Update on 16/03/2012. Changed the Win64 OpenSSL v1.0.1 Light tools.

Update 27/03/2012. Added a additional subjectAltName to the Certificate request. But my had my parameters wrong.

Update (27/03/2012): I have added a new subjectAltName on the to my openssl.cfg. I added the FQDN name of my vCenter server in the Certificate request. With vCenter Operations Manager 5.0, you get the integration within the vCenter Client in the Solutions & Applications section. The SSL Certificates will therefore be checked by the vCenter Client against the vCenter FQDN name.

Update 29/03/2012. Thanks for Kinsei for having raises the question on the topic of the SSL Certificate usage via the vCenter Client. The vCenter Operations Manager is connected to the vCenter Server not by an FQDN name, but by an IP Address. You can find the value when you connect to your vCenter server’s Managed Object Reference (mob) settings portal.

https://vcenter/mob/ Content ExtensionManager ExtensionList com.vmware.vcops

 Update (02/04/2012). Here is another update. Josh Perkins has written up a great article on how to ensure that your vCenter uses a FQDN or shortname to speak to your vCenter Operations Manager. This means that administrators and user on the vSphere Client do not get invalid SSL Certificate requests anymore. Thanks Josh !!

 

First Steps with PowerCLI

 

When starting to use PowerCLI 5.0.1 for the first time, there are a few things that will help you start off a good foot. I for one, have decided when installing the PowerCLI on my system to modify the installation to add the vCloud Director PowerCLI feature and to change the path of installation, so that I can quickly find my .PS1 on my development machine.

Installing PowerCLI 5.0.1 Build 4431

Selecting vCloud Director PowerCLI feature & changing install Path

Once installed on my development machine, I add start the PowerCLI to my taskbar and start it. I then modify the Layout settings to get a better usable window.

Modifying PowerCLI Layout

And now we are getting to the two basic commands you add to your PowerCLI to ensure you can run RemoteSigned code.

[box] set-ExecutionPolicy RemoteSigned[/box]

Set-ExecutionPolicy RemoteSigned

As I’m developping my code on a system other than the vCenter Server. I will get SSL Certificates warnings if I remotely connect to my vCenter. To ignore the Certificate warnings I use to following command

[box] Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -WarningAction SilentlyContinue[/box]

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -WarningAction SilentlyContinue

When connecting to my vCenter I now only get a pop-uo for the User Credentials.

Connect-VIServer will request Credentials

I can also save my credentials for future use.

Connect-VIServer with Credentials saving

And so the next time I don’t even need to re-enter these credentials.

Connect-VIServer using saved Credentials