Released: VMware ESXi 4.1 Update 1

February 13, 2011

VMware has released a new update of the flagship product VMware ESXi. VMware ESXi 4.1 update 1 has been released and following information describes some of the enhancements available in this release of VMware ESXi:

  • Enablement of Trusted Execution Technology (TXT) ESXi 4.1 Update 1 can be configured to boot with Intel Trusted Execution Technology (TXT). This boot option can protect ESXi in some cases where system binaries are corrupted or have been tampered with. TXT is currently available on Intel Xeon processor 5600 series servers. For more information, see KB 1033811.
  • Improvement in scalability — ESXi 4.1 Update 1 supports up to 160 logical processors.
  • Support for additional guest operating systems ESXi 4.1 Update 1 provides support for RHEL 6, RHEL 5.6, SLES 11 SP1 for VMware, Ubuntu 10.10, and Solaris 10 Update 9 guest operating systems. For a complete list of guest operating systems supported in this release, see the VMware Compatibility Guide.
  • Inclusion of additional drivers ESXi 4.1 Update 1 includes the 3ware SCSI and Neterion vxge drivers. For earlier releases, these drivers are only available as separate downloads.

ESX/ESXi 4.1 with Broadcom bnx2x experiences a purple diagnostic screen

October 13, 2010

During an implementation of ESXi 4.1 at a customer site I experienced random crashes of ESXi during normal operations. VMware support has been investigating this issue and a probem has been found in the Broadcom’s bnx2x Inbox driver version 1.54.1.v41.1-1vmw. This issue can lead to a complete crash of the ESX/ESXi 4.1 host.

VMware has currently released a workaround for this bug and is working to provide a full solution to the issue. Information on this problem can be found in KB1029368.

Application consistent quiescing and VDR

August 18, 2010

Taking snapshots of your virtual machines seems an adequate way of protecting your virtual machine and it application data. However, protecting transactional systems like SQL, Active Directory, Exchange and other systems using snapshots requires a little more attention.

To create a working snapshot of your virtual machine, the virtual machine needs to be in a Quiesce state in order to guarantee a consistent and usable backup. This generally requires flushing any outstanding writes.

VMware Data Recovery (VDR) uses Microsoft Windows Volume Shadow Copy Service (VSS) quiescing, which provides the backup infrastructure for certain Windows operating systems, as well as a mechanism for creating consistent point-in-time copies of data known as shadow copies.

VSS produces consistent shadow copies by coordinating with business applications, file-system services, backup applications, fast-recovery solutions, and storage hardware. VSS support is provided with VMware Tools, which runs in the guest operating system. Until VMware vSphere 4.1, VMware did not fully support the VSS-writers if you are running Windows Server 2008 virtual machines. A good friend of mine has an excellent article on this.

With VMware vSphere 4.1 this has changed and VMware now fully supports application level quiescing through VSS. VMware Data Recovery uses different quiescing mechanisms depending on the guest operating system that you run in your virtual machines. See the table below for the quiescing types.


For applicationconsistent quiescing to be available, three conditions  must be met:

  • The UUID attribute must be enabled. This is enabled by default on virtual machines created on ESX 4.1 hosts.
  • The virtual machine must use only SCSI disks. For example, application-consistent quiescing is not supported for virtual machines with IDE disks. There must as many free SCSI slots in the virtual machine as the number of disks. For example, if there are 8 SCSI disks on SCSI adapter 1, there are not enough SCSI slots free to perform application quiescing.
  • The virtual machine must not use dynamic disks.

Windows 2008 virtual machines created on ESX/ESXi 4.0 hosts can be enabled for application consistent quiescing on ESX/ESXi 4.1 hosts by enabling the disk UUID attribute. Use the following procedure to enable the disk UUID:

  1. Start the vSphere Client, and log in to a vCenter Server.
  2. Select Virtual Machines and Templates and click the Virtual Machines tab.
  3. Right-click the Windows 2008 virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off.
    The virtual machine powers off.
  4. Right-click the virtual machine, and click Edit Settings.
  5. Click the Options tab, and select the General entry in the settings column.
  6. Click Configuration Parameters
    The Configuration Paramters window appears.
  7. Click Add Row.
  8. In the Name column, enter disk.EnableUUID.
  9. In the Value column, enter TRUE.
  10. Click OK and click Save.
  11. Power on the virtual machine.

Application consistent quiescing is available for this virtual machine now that the UUID property has been enabled.

VMware ESX 4.0 Update 2

June 14, 2010

VMware has released the next major update on VMware ESX 4.0 with the release of VMware ESX 4.0 Update 2.

The following information provides highlights of some of the enhancements available in this release of VMware ESX:

  • Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.
  • Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.
  • Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.
  • Enhancement of the esxtop/resxtop utility— vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).
  • Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Resolved Issues In addition, this release delivers a number of bug fixes that have been documented in the Resolved Issues section.

The full release notes forVMware ESX 4.0 Update 2 can be read here.

VMware ESX 4.0 Update 2 can be downloaded here.

Texas Memory Systems RamSan-620

April 27, 2010

I am currently evaluating a 2 TB RamSan-620 SSD storage array from Texas Memory Systems attached to a vSphere 4 infrastructure. The RamSan-620 is a rack-mounted SLC NAND Flash SSD that provides plenty of shareable, high performance storage. The RamSan-620 has extremely low latency and should delivers 250,000 IOPS of sustained performance according to specifications.

The RamSan-620 is not on the HCL of VMware, but it worked with all the defined technical tests and all features of vSphere 4. So after verifying the unit worked correctly with vSphere it’s time to test the performance of the device. The RamSan-620 is equipped with two dualport 4GB fibre channel controllers and can be expanded with two additional controllers, resulting in eight fibre channel ports.

In the test setup the RamSan-620 is connected to two HP BL460 G6 blades through two HP Brocade 8GB SAN switches. Each blade has a QLogic QMH2462 4Gb dual port FC HBA and runs ESXi 4 update 1. The quedepth of the QMH2462 has been adjusted to 256.


Each ESXi 4 server contains a Windows 2003 virtual machine with a VMware Paravirtual SCSI (PVSCSI) adapter, 4 vCPU’s and runs in a datastore on a RamSan-620 lun. The lun’s are configured with the VMware round-robin path policy.

I used the SQLIO performance tool from Microsoft to perform various tests on the unit. To test the I/O throughput I fired up the following SQLIO command on both virtual machines:


This simulated four threads performing 4KB random reads on 2 GB files. Each VM performs around 60218 4KB random reads resulting in a total 120436 IOPS. During this tests all vCPU’s run at 100% utilization. 99% of all reads have an average latency of 0 ms. Very good results!

On the RamSan-620 fibrechannel interfaces we can see that the IO load is equally spread among all interfaces.


Using ATTO Disk benchmark I tested the throughput by starting a benchmark on one virtual machine. With 8192 KB blocks reads a throughput of 800 MB/sec is reached. In this case the 4-Gigabit HBA of the blade server becomes the bottleneck. The 4-Gigabit HBA delivers 800 MB/sec throughput per channel in full-duplex mode. Since ESX 4 always uses one active path with round-robin path policy, a maximum of 800 MB/sec throughput can be reached.


From these basic performance tests the RamSan-620 SSD storage device proves the be a high performance storage with extremely low latency and is very useful as a tier 0 storage device for a VMware vSphere Infrastructure environment. In the next weeks I will further test the device and post my results.

Adjust Queue Depth on vSphere ESXi

March 23, 2010

The procedure for changing the queue depth of a HBA on vSphere ESXi servers differs a little bit from vSphere ESX servers. For ESXi you can performe this procedure from the vSphere CLI or the vSphere Management Assistant (vMA).

The following example describes the procedure for changing the queue depth for a QLogic adapter:

  1. Verify which QLogic HBA module is currently loaded by entering the following command:esxcfg-module -list | grep qla
  2. Run the following commands. The example shows the qla2xxx module. Use the appropriate module based on the outcome of the previous step.vicfg-module -s ql2xmaxqdepth=64 qla2xxx

    In this case, the HBA represented by qla2xxx will have its LUN queue depth set to 64.

  3. Reboot your host.

Common VMkernel errors

February 10, 2010

In a lot of ESX/ESXi implementations I noticed some common warnings messages in the VMkernel log. I wanted to get more insight into these warnings so I contacted VMware support for some explanation. Here are my results:

WARNING: UserCartel: 1820: Fork not supported for multithreaded parents
This message is normally caused by a known bug in qlogic drivers. VMware can’t provide an exact cause as they are still waiting for qlogic to come back with an updated driver but there is no cause for concern, no impact on the system has been demonstrated.

WARNING: UserLinux: 1717: UNIMPLEMENTED! write-back of mmap regions unsupported
This message is caused problem in the CIM agent in combination with the OEM providers and some IHV information that is reported whether the Hardware exists or not on the host. This problem can result in the error generated. The message can be ignored for the moment and will be fixed in a future patch.

WARNING: VFAT: 154: File_Ioctl
The messages is not a cause for concern. There messages are being created by busybox. Despite the ‘warning’ tag they are actually information and are being generated as busybox interacts with parts of the filesystem. There is an upcoming patch due which will stop the messages from being logged.