Common VMkernel errors

In a lot of ESX/ESXi implementations I noticed some common warnings messages in the VMkernel log. I wanted to get more insight into these warnings so I contacted VMware support for some explanation. Here are my results:

WARNING: UserCartel: 1820: Fork not supported for multithreaded parents
This message is normally caused by a known bug in qlogic drivers. VMware can’t provide an exact cause as they are still waiting for qlogic to come back with an updated driver but there is no cause for concern, no impact on the system has been demonstrated.

WARNING: UserLinux: 1717: UNIMPLEMENTED! write-back of mmap regions unsupported
This message is caused problem in the CIM agent in combination with the OEM providers and some IHV information that is reported whether the Hardware exists or not on the host. This problem can result in the error generated. The message can be ignored for the moment and will be fixed in a future patch.

WARNING: VFAT: 154: File_Ioctl
The messages is not a cause for concern. There messages are being created by busybox. Despite the ‘warning’ tag they are actually information and are being generated as busybox interacts with parts of the filesystem. There is an upcoming patch due which will stop the messages from being logged.

P2V Essentials

I am currently doing a P2V conversion for one of my customers. Cleaning up the converted server can be a time consuming job. In this previous post I mentioned two tools to remove HP management software from the converted server.

The P2V conversion process also leaves a lot of inactive devices in the device manager. Removing these devices is a lot of work.

Rob van der Woude has created a great script based on the Microsoft Tool ‘devcon’, to automatically remove all inactive devices. This script is a valuable tool for your P2V conversion process!

You can download the script here.

Virtualizing Citrix XenApp on vSphere, part II

 

This is the second part of my articles on virtualizing XenApp on vSphere. You can read the first part here.

vSphere design considerations

I recommend to create a dedicated VMware cluster for virtualized XenApp servers. The reason for this is use of vSphere functionality and the resulting licensing costs. Because XenApp VM’s are highly resource intensive there is no need for VMware DRS. VMware DRS is useful when there are a lot of VM’s with different resource demands but with XenApp VM’s this is not the case. All XenApp VM’s are resource intensive. Furthermore I do not recommend to use vmotion on XenApp VM’s because end-users could experience a delay in there XenApp session during a vmotion. Based on these facts a vSphere Standard edition license gives enough functionality for the VMware cluster with XenApp servers, which I will call the VMware XenApp cluster in the rest of this article. By using vSphere standard, the XenApp servers are protected against hardware failures by VMware HA and using the vSphere standard license keeps the cost of a virtualized XenApp server low. There’s one caveat, if you use distributed vSwitches in your environment you are stuck with standard vSwitches with vSphere standard or you should use the vSphere Enterprise Plus license for the VMware XenApp cluster.

Hardware considerations

Selecting the right server hardware is essential for successful virtualization of XenApp servers. I recommend to use hardware based on the Intel Nehalem processor architecture. At time of writing this article, this is the Intel Xeon 55XX series. The Nehalem processor architecture provides support for hardware-assisted memory management unit (MMU) virtualization. MMU virtualization provides a significant performance increase.

The number of physical CPU cores should be matched with the number of vCPU’s. For virtualized XenApp servers, do not overcommit CPU resources. For example, if you intend to run 4 XenApp servers with 2 vCPU’s, your ESX server should have 8 cores. It is possible to slightly overcommit but you should monitor your CPU ready time for bottlenecks.

I used diskless HP Proliant BL460 G6 servers for building a VMware XenApp cluster, but any modern server will do. These servers are equipped with 2 Quadcore Intel E5540 CPU’s and 24 GB of RAM. With this configuration I managed to run 5 XenApp VM’s with a good performance. With 6 XenApp VM’s the CPU ready time becomes greater than 5% and the performance isn’t acceptable anymore.

Because XenApp VM’s are disk I/O intensive (especially during logon hours) I recommend to make dedicated datastores with the size for holding a maximum of four XenApp VM’s within the VMware XenApp cluster. This way the XenApp’s VM’s make optimal use of the I/O queue’s each lun provide. Also create the datastores on the fastest lun you can provide. I used raid 1 fibrechannel lun’s for the XenApp VM datastores.

Application Landscape

Each XenApp VM will contain some standard applications and custom application specific to the business. In this article I focus only on the standard applications. Here’s a quick overview of the standard application set I use on a XenApp VM:

- Microsoft Internet Explorer 7
- Microsoft Office 2003
- Antivirus software

I used Internet Explorer 7 because Internet Explorer 8 had more CPU spikes. I don’t know the reason for this, maybe some web application has not been optimized for IE8 but I didn’t have the time to investigate this problem. By default IE8 also starts two iexplorer processes for each user which requires a little more resources. Office 2003 was used because the customer was still using Office 2003. This is a plus because Office 2007 requires more resources. I recommended to use antivirus software on your XenApp VM.

With this configuration I managed to run 5 XenApp VM’s on a single ESXi host with each XenApp VM supporting 25 users. This makes a total of 150 users on one physical server. The user experience with this configuration is within normal range.

The following chart show’s the CPU usage from the ESXi host with 5 XenApp servers running.

The following chart show’s the CPU usage from a XenApp VM.

Please note that the CPU ready time is below the 5%.

vSphere Storage Bug

A bug has been found concerning the vStorage system of vSphere 4. Removing a lun from a vSphere 4 cluster could make all datastores unavailable. This bug can occur if you do not remove the lun and datastore form the VI Client before removing the lun from the array and/or fabric.

More information can be found in the following blog post:

http://virtualgeek.typepad.com/virtual_geek/2009/12/an-important-vsphere-4-storage-bug-and-workaround.html

vKernel Capacity Modeler available for free

With the release of VMware vCenter CapacityIQ, VMware has provided the functionality to vCenter to proactively monitor the capacity of your VMware Infrastructure. It’s great that VMware recognizes the market need for capacity planning, management, and optimization of VMware infrastructures. However the VMware vCenter CapacityIQ is quite expensive and a first release for VMware.

vKernel has a similar product for quite some time with lots of features under the name of vKernel Capacity Analyzer.

vKernel Corporation is making perpetual licenses of its Capacity Modeler software for VMware completely free until December 31, 2009. Your free license for vKernel Capacity Modeler can be downloaded here:

http://www.vkernel.com/free

Virtualizing Citrix XenApp on vSphere, part I

Today many organizations who use XenApp are still bound to x86 platforms because of legacy applications which don’t run on a x64 platform. Sometimes an application does run on a x64 platform but is not supported by the vendor on a x64 platform. By sticking to the x86 platform, modern server hardware can’t be fully utilized due to the memory limit of 4 GB of the x86 platform.

To overcome this limitation virtualization comes to practice! Virtualizing a XenApp server has always been a challenge. However with the maturity of vSphere and current CPU’s, there isn’t a limitation anymore to not virtualize a XenApp server. In this article I will share you my experience of implementing a virtualized XenApp production environment for a customers and give you my recommendations for a successful virtualization of XenApp.

This article is split in different parts. In the first part I focus on the configuration of the XenApp VM. In the second part I will look at the application landscape and the underlying ESX host and in the last part I will look at the performance results.

Building a XenApp VM

The configuration of a XenApp VM is crucial to the performance of a virtualized XenApp server and has a direct link with the hardware configuration of the underlying ESX host. As with every VM there are four main performance aspects that need to be optimized for the corresponding workload: cpu, memory, disk I/O and network I/O.

CPU
The most important resource for the XenApp VM is CPU. XenApp VM’s tend to use a lot of CPU resources and this is most likely to be the first bottleneck. In creating you XenApp VM, there are two scenario’s: scale-out or scale-up. In the scale-out scenario there a lot of light XenApp VM’s created with one vCPU. In the scale-up scenario less VM’s are created with two vCPU. The main objective is to not over commit your physical CPU’s. Let’s say you have a ESX host with two Quadcore CPU’s, which is a total of 8 cores. If you create eight 1 vCPU VM’s, each VM can schedule a dedicated CPU. The same applies for 2 vCPU VM’s, if you create four 2 vCPU VM’s, each VM can schedule a dedicated set of CPU’s.

Depending of the workload on your XenApp VM, one of this scenario’s fits best. If you have light workloads the scale-out scenario might be best, but in most situations the scale-up scenario does the best job. In most circumstances, using 2 vCPU’s allows for more users and a more consistent user experience. With the improvements made to the CPU scheduler in vSphere, like further relaxed co-scheduling, SMP XenApp VM’s are no longer a problem. If you are using a host with Intel Nehalem CPU’s enable the CPU and Memory hardware assistance (more information on this in part II).

Disk I/O
The second important resource is disk I/O. This will be further explained in the next part of this article but for now I recommend to use two virtual disks for a XenApp VM. One for the operating system and one for the applications. For optimal disk I/O performance, make sure you align the file system in the guest OS.

Memory
The next resource for the XenApp VM is memory. With memory there is one simple rule. Don´t over commit memory for your XenApp VM´s. Depending of the workload of the XenApp server, configure the XenApp VM with the corresponding amount of memory. In most situations this will be 4096 MB of RAM (assuming you are using a 32 bit OS). Make sure you also make a memory reservation of the same size. This way the XenApp VM has all the configured RAM available and the VMware balloon driver cannot degrade the performance of the XenApp VM.

Network I/O
The last resource for the XenApp VM is network. I haven´t seen any XenApp VM implementation where network i/o results in a bottleneck but for best results use the new VMXNET 3 vNIC. The VMXNET 3 has less impact on the CPU which is always useful.

Other considerations
I recommend to use Windows Server 2003 R2 x86 for building the XenApp VM. Windows Server 2008 uses a lot more resources. This probably will be a lot better with Windows Server 2008 R2 but at time of writing this article, XenApp is not certified for use with Windows Server 2008 R2. Furthermore I recommend to remove the CDRom drive and floppy disks. The floppy disk can be complete disabled in the BIOS of the VM. Always install the latest VMware Tools to provide the optimized drivers for the virtual SCSI and network devices and install the balloon driver.

So let’s summarize the preferable XenApp VM configuration:

2 vCPU
4096 MB of RAM with 4096 MB reserved
1 LSI Logic Parallel SCSI controller
2 virtual disks, one of OS and one for applications
1 VMXNET 3 vNIC
CDRom drive removed
Floppy drive removed
CPU and Memory hardware assistance enabled if using Intel Nehalim processors

Again, depending on your environment another configuration could be more desirable. A consistent server configuration is very important in a XenApp farm so I recommend to build a dedicated template for deploying XenApp VM’s.

In the next part of this article I will look at the application landscape and the underlying ESX host for building your virtualized XenApp farm.

Continue to part II.

VMware vCenter CapacityIQ has been released

VMware vCenter CapacityIQ has been released by VMware. vCenter CapacityIQ is vCenter plugin for proactively monitor the capacity of your VMware Infrastructure. vCenter CapacityIQ monitors cpu, memory and disk I/O usage and predicts future demands for resources.

vCenter CapacityIQ allows you to show impacts in capacity as well as forecasted capacity based on changes you predict will happen to the environment.

From the VMware site:

‘VMware vCenter CapacityIQ is a value-add component of the vCenter family of management solutions, providing capacity management capabilities for virtualized datacenter or desktop environments. CapacityIQ integrates seamlessly with VMware vCenter Server, ensuring that your virtualized infrastructure capacity is always predictable and efficiently used.

• Eliminate waste by identifying any unused or over-allocated capacity
• Reduce operational overhead by automating routine capacity monitoring and management tasks
• Minimize business risk of outages or failures resulting from capacity bottlenecks and shortfalls

More information can be found on: http://www.vmware.com/products/vcenter-capacityiq/

vSphere, HP EVA and path policies

The HP EVA comes in two types of storage arrays. An active/passive array and an active/active array. The HP EVA active/active storage arrays can handle I/O requests on both storage processors. With ESX 3.5 and an active/active EVA array you had two options for the path selection policy, fixed and MRU (Most Recently Used). With an active/active EVA array most people used the fix path policy to manually load balance the I/O over both storage processors. This process is very work intensive (it has to be done on all ESX hosts) and is prone to errors (e.g. a fixed path is configured to the wrong storage processor or path trashing).

With the release of vSphere and the vStorage API’s for multipathing, the round robin path selection policy is now fully supported. The HP EVA active/active storage arrays support ALUA which stands for Asymmetrical Logical Unit Access. ALUA occurs when the access characteristics of one port may differ from those of another port. ALUA allows a lun to be accessed via its primary path (via the owning Storage Processor) and via an asymmetrical path (via the not-owning Storage Processor). I/O to the not-owning Storage Processor or not-optimized path  comes with a performance penalty because the I/O has to be transmitted over the internal connection between the storage processors which does not have much bandwidth.

With ALUA the ESX host is aware of the non-optimized paths. So when you use the round robin path policy, ESX will load balance the I/O over the optimized paths (the paths to the owning storage processor of the lun) and use the non-optimized paths only in case of a failure on the optimized paths. To my opinion this has two advances: both path are used and manually load balancing isn’t necessary anymore which saves a lot of configuration work.

So here you have it! I recommend to use the round robin path policy with an active/active EVA array. If you don’t want to use the round robin path policy I would recommend to use the MRU policy. The MRU policy is ALUA aware and will also use the optimized paths.

XenApp on vSphere

If you’re a XenApp user making a virtualization platform choice you need to read the new Project VRC paper that came out last week NOW.  The paper is titled, "VRC, VSI and Clocks Reviewed".  The results have shifted dramatically since the January findings.  Instead of trailing, ESX 4.0 now has a 4% advantage over XenServer 5.5 in the number of Terminal Services users it can support on a server. 

You can read the VRC whitepaper here (requires free registration).

New vSphere patches released

New patches have been released for vSphere:

VMware ESX 4.0, Patch Release ESX400-200909001 (1014078)
Date Published: 9/25/2009
VMware ESXi 4.0, Patch Release ESXi400-200909001 (1014079)
Date Published: 9/25/2009
VMware ESX 4.0, Patch Release and Cisco Nexus 1000V Virtual Ethernet Module (1014080)
Date Published: 9/25/2009
VMware ESXi 4.0, Patch Release and Cisco Nexus 1000V Virtual Ethernet Module (1014081)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909401-BG: Updates vmx and vmkernel64 (1014019)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909402-BG: Updates VMware Tools (1014020)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909403-BG: Updates bnx2x driver (1014021)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909404-BG: Updates ixgbe driver (1014022)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909405-BG: Updates perftools (1014023)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909406-BG: Updates hpsa driver (1014024)