ESX/ESXi 4.1 with Broadcom bnx2x experiences a purple diagnostic screen

October 13, 2010

During an implementation of ESXi 4.1 at a customer site I experienced random crashes of ESXi during normal operations. VMware support has been investigating this issue and a probem has been found in the Broadcom’s bnx2x Inbox driver version 1.54.1.v41.1-1vmw. This issue can lead to a complete crash of the ESX/ESXi 4.1 host.

VMware has currently released a workaround for this bug and is working to provide a full solution to the issue. Information on this problem can be found in KB1029368.


Application consistent quiescing and VDR

August 18, 2010

Taking snapshots of your virtual machines seems an adequate way of protecting your virtual machine and it application data. However, protecting transactional systems like SQL, Active Directory, Exchange and other systems using snapshots requires a little more attention.

To create a working snapshot of your virtual machine, the virtual machine needs to be in a Quiesce state in order to guarantee a consistent and usable backup. This generally requires flushing any outstanding writes.

VMware Data Recovery (VDR) uses Microsoft Windows Volume Shadow Copy Service (VSS) quiescing, which provides the backup infrastructure for certain Windows operating systems, as well as a mechanism for creating consistent point-in-time copies of data known as shadow copies.

VSS produces consistent shadow copies by coordinating with business applications, file-system services, backup applications, fast-recovery solutions, and storage hardware. VSS support is provided with VMware Tools, which runs in the guest operating system. Until VMware vSphere 4.1, VMware did not fully support the VSS-writers if you are running Windows Server 2008 virtual machines. A good friend of mine has an excellent article on this.

With VMware vSphere 4.1 this has changed and VMware now fully supports application level quiescing through VSS. VMware Data Recovery uses different quiescing mechanisms depending on the guest operating system that you run in your virtual machines. See the table below for the quiescing types.


For applicationconsistent quiescing to be available, three conditions  must be met:

  • The UUID attribute must be enabled. This is enabled by default on virtual machines created on ESX 4.1 hosts.
  • The virtual machine must use only SCSI disks. For example, application-consistent quiescing is not supported for virtual machines with IDE disks. There must as many free SCSI slots in the virtual machine as the number of disks. For example, if there are 8 SCSI disks on SCSI adapter 1, there are not enough SCSI slots free to perform application quiescing.
  • The virtual machine must not use dynamic disks.

Windows 2008 virtual machines created on ESX/ESXi 4.0 hosts can be enabled for application consistent quiescing on ESX/ESXi 4.1 hosts by enabling the disk UUID attribute. Use the following procedure to enable the disk UUID:

  1. Start the vSphere Client, and log in to a vCenter Server.
  2. Select Virtual Machines and Templates and click the Virtual Machines tab.
  3. Right-click the Windows 2008 virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off.
    The virtual machine powers off.
  4. Right-click the virtual machine, and click Edit Settings.
  5. Click the Options tab, and select the General entry in the settings column.
  6. Click Configuration Parameters
    The Configuration Paramters window appears.
  7. Click Add Row.
  8. In the Name column, enter disk.EnableUUID.
  9. In the Value column, enter TRUE.
  10. Click OK and click Save.
  11. Power on the virtual machine.

Application consistent quiescing is available for this virtual machine now that the UUID property has been enabled.

VMware ESX 4.0 Update 2

June 14, 2010

VMware has released the next major update on VMware ESX 4.0 with the release of VMware ESX 4.0 Update 2.

The following information provides highlights of some of the enhancements available in this release of VMware ESX:

  • Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.
  • Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.
  • Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.
  • Enhancement of the esxtop/resxtop utility— vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).
  • Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Resolved Issues In addition, this release delivers a number of bug fixes that have been documented in the Resolved Issues section.

The full release notes forVMware ESX 4.0 Update 2 can be read here.

VMware ESX 4.0 Update 2 can be downloaded here.

Texas Memory Systems RamSan-620

April 27, 2010

I am currently evaluating a 2 TB RamSan-620 SSD storage array from Texas Memory Systems attached to a vSphere 4 infrastructure. The RamSan-620 is a rack-mounted SLC NAND Flash SSD that provides plenty of shareable, high performance storage. The RamSan-620 has extremely low latency and should delivers 250,000 IOPS of sustained performance according to specifications.

The RamSan-620 is not on the HCL of VMware, but it worked with all the defined technical tests and all features of vSphere 4. So after verifying the unit worked correctly with vSphere it’s time to test the performance of the device. The RamSan-620 is equipped with two dualport 4GB fibre channel controllers and can be expanded with two additional controllers, resulting in eight fibre channel ports.

In the test setup the RamSan-620 is connected to two HP BL460 G6 blades through two HP Brocade 8GB SAN switches. Each blade has a QLogic QMH2462 4Gb dual port FC HBA and runs ESXi 4 update 1. The quedepth of the QMH2462 has been adjusted to 256.


Each ESXi 4 server contains a Windows 2003 virtual machine with a VMware Paravirtual SCSI (PVSCSI) adapter, 4 vCPU’s and runs in a datastore on a RamSan-620 lun. The lun’s are configured with the VMware round-robin path policy.

I used the SQLIO performance tool from Microsoft to perform various tests on the unit. To test the I/O throughput I fired up the following SQLIO command on both virtual machines:


This simulated four threads performing 4KB random reads on 2 GB files. Each VM performs around 60218 4KB random reads resulting in a total 120436 IOPS. During this tests all vCPU’s run at 100% utilization. 99% of all reads have an average latency of 0 ms. Very good results!

On the RamSan-620 fibrechannel interfaces we can see that the IO load is equally spread among all interfaces.


Using ATTO Disk benchmark I tested the throughput by starting a benchmark on one virtual machine. With 8192 KB blocks reads a throughput of 800 MB/sec is reached. In this case the 4-Gigabit HBA of the blade server becomes the bottleneck. The 4-Gigabit HBA delivers 800 MB/sec throughput per channel in full-duplex mode. Since ESX 4 always uses one active path with round-robin path policy, a maximum of 800 MB/sec throughput can be reached.


From these basic performance tests the RamSan-620 SSD storage device proves the be a high performance storage with extremely low latency and is very useful as a tier 0 storage device for a VMware vSphere Infrastructure environment. In the next weeks I will further test the device and post my results.

Common VMkernel errors

February 10, 2010

In a lot of ESX/ESXi implementations I noticed some common warnings messages in the VMkernel log. I wanted to get more insight into these warnings so I contacted VMware support for some explanation. Here are my results:

WARNING: UserCartel: 1820: Fork not supported for multithreaded parents
This message is normally caused by a known bug in qlogic drivers. VMware can’t provide an exact cause as they are still waiting for qlogic to come back with an updated driver but there is no cause for concern, no impact on the system has been demonstrated.

WARNING: UserLinux: 1717: UNIMPLEMENTED! write-back of mmap regions unsupported
This message is caused problem in the CIM agent in combination with the OEM providers and some IHV information that is reported whether the Hardware exists or not on the host. This problem can result in the error generated. The message can be ignored for the moment and will be fixed in a future patch.

WARNING: VFAT: 154: File_Ioctl
The messages is not a cause for concern. There messages are being created by busybox. Despite the ‘warning’ tag they are actually information and are being generated as busybox interacts with parts of the filesystem. There is an upcoming patch due which will stop the messages from being logged.

Virtualizing Citrix XenApp on vSphere, part I

October 26, 2009

Today many organizations who use XenApp are still bound to x86 platforms because of legacy applications which don’t run on a x64 platform. Sometimes an application does run on a x64 platform but is not supported by the vendor on a x64 platform. By sticking to the x86 platform, modern server hardware can’t be fully utilized due to the memory limit of 4 GB of the x86 platform.

To overcome this limitation virtualization comes to practice! Virtualizing a XenApp server has always been a challenge. However with the maturity of vSphere and current CPU’s, there isn’t a limitation anymore to not virtualize a XenApp server. In this article I will share you my experience of implementing a virtualized XenApp production environment for a customers and give you my recommendations for a successful virtualization of XenApp.

This article is split in different parts. In the first part I focus on the configuration of the XenApp VM. In the second part I will look at the application landscape and the underlying ESX host and in the last part I will look at the performance results.

Building a XenApp VM

The configuration of a XenApp VM is crucial to the performance of a virtualized XenApp server and has a direct link with the hardware configuration of the underlying ESX host. As with every VM there are four main performance aspects that need to be optimized for the corresponding workload: cpu, memory, disk I/O and network I/O.

The most important resource for the XenApp VM is CPU. XenApp VM’s tend to use a lot of CPU resources and this is most likely to be the first bottleneck. In creating you XenApp VM, there are two scenario’s: scale-out or scale-up. In the scale-out scenario there a lot of light XenApp VM’s created with one vCPU. In the scale-up scenario less VM’s are created with two vCPU. The main objective is to not over commit your physical CPU’s. Let’s say you have a ESX host with two Quadcore CPU’s, which is a total of 8 cores. If you create eight 1 vCPU VM’s, each VM can schedule a dedicated CPU. The same applies for 2 vCPU VM’s, if you create four 2 vCPU VM’s, each VM can schedule a dedicated set of CPU’s.

Depending of the workload on your XenApp VM, one of this scenario’s fits best. If you have light workloads the scale-out scenario might be best, but in most situations the scale-up scenario does the best job. In most circumstances, using 2 vCPU’s allows for more users and a more consistent user experience. With the improvements made to the CPU scheduler in vSphere, like further relaxed co-scheduling, SMP XenApp VM’s are no longer a problem. If you are using a host with Intel Nehalem CPU’s enable the CPU and Memory hardware assistance (more information on this in part II).

Disk I/O
The second important resource is disk I/O. This will be further explained in the next part of this article but for now I recommend to use two virtual disks for a XenApp VM. One for the operating system and one for the applications. For optimal disk I/O performance, make sure you align the file system in the guest OS.

The next resource for the XenApp VM is memory. With memory there is one simple rule. Don´t over commit memory for your XenApp VM´s. Depending of the workload of the XenApp server, configure the XenApp VM with the corresponding amount of memory. In most situations this will be 4096 MB of RAM (assuming you are using a 32 bit OS). Make sure you also make a memory reservation of the same size. This way the XenApp VM has all the configured RAM available and the VMware balloon driver cannot degrade the performance of the XenApp VM.

Network I/O
The last resource for the XenApp VM is network. I haven´t seen any XenApp VM implementation where network i/o results in a bottleneck but for best results use the new VMXNET 3 vNIC. The VMXNET 3 has less impact on the CPU which is always useful.

Other considerations
I recommend to use Windows Server 2003 R2 x86 for building the XenApp VM. Windows Server 2008 uses a lot more resources. This probably will be a lot better with Windows Server 2008 R2 but at time of writing this article, XenApp is not certified for use with Windows Server 2008 R2. Furthermore I recommend to remove the CDRom drive and floppy disks. The floppy disk can be complete disabled in the BIOS of the VM. Always install the latest VMware Tools to provide the optimized drivers for the virtual SCSI and network devices and install the balloon driver.

So let’s summarize the preferable XenApp VM configuration:

2 vCPU
4096 MB of RAM with 4096 MB reserved
1 LSI Logic Parallel SCSI controller
2 virtual disks, one of OS and one for applications
CDRom drive removed
Floppy drive removed
CPU and Memory hardware assistance enabled if using Intel Nehalim processors

Again, depending on your environment another configuration could be more desirable. A consistent server configuration is very important in a XenApp farm so I recommend to build a dedicated template for deploying XenApp VM’s.

In the next part of this article I will look at the application landscape and the underlying ESX host for building your virtualized XenApp farm.

Continue to part II.

New vSphere patches released

September 28, 2009

New patches have been released for vSphere:

VMware ESX 4.0, Patch Release ESX400-200909001 (1014078)
Date Published: 9/25/2009
VMware ESXi 4.0, Patch Release ESXi400-200909001 (1014079)
Date Published: 9/25/2009
VMware ESX 4.0, Patch Release and Cisco Nexus 1000V Virtual Ethernet Module (1014080)
Date Published: 9/25/2009
VMware ESXi 4.0, Patch Release and Cisco Nexus 1000V Virtual Ethernet Module (1014081)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909401-BG: Updates vmx and vmkernel64 (1014019)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909402-BG: Updates VMware Tools (1014020)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909403-BG: Updates bnx2x driver (1014021)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909404-BG: Updates ixgbe driver (1014022)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909405-BG: Updates perftools (1014023)
Date Published: 9/25/2009
VMware ESX 4.0, Patch ESX400-200909406-BG: Updates hpsa driver (1014024)