Texas Memory Systems RamSan-620

April 27, 2010

I am currently evaluating a 2 TB RamSan-620 SSD storage array from Texas Memory Systems attached to a vSphere 4 infrastructure. The RamSan-620 is a rack-mounted SLC NAND Flash SSD that provides plenty of shareable, high performance storage. The RamSan-620 has extremely low latency and should delivers 250,000 IOPS of sustained performance according to specifications.

The RamSan-620 is not on the HCL of VMware, but it worked with all the defined technical tests and all features of vSphere 4. So after verifying the unit worked correctly with vSphere it’s time to test the performance of the device. The RamSan-620 is equipped with two dualport 4GB fibre channel controllers and can be expanded with two additional controllers, resulting in eight fibre channel ports.

In the test setup the RamSan-620 is connected to two HP BL460 G6 blades through two HP Brocade 8GB SAN switches. Each blade has a QLogic QMH2462 4Gb dual port FC HBA and runs ESXi 4 update 1. The quedepth of the QMH2462 has been adjusted to 256.


Each ESXi 4 server contains a Windows 2003 virtual machine with a VMware Paravirtual SCSI (PVSCSI) adapter, 4 vCPU’s and runs in a datastore on a RamSan-620 lun. The lun’s are configured with the VMware round-robin path policy.

I used the SQLIO performance tool from Microsoft to perform various tests on the unit. To test the I/O throughput I fired up the following SQLIO command on both virtual machines:


This simulated four threads performing 4KB random reads on 2 GB files. Each VM performs around 60218 4KB random reads resulting in a total 120436 IOPS. During this tests all vCPU’s run at 100% utilization. 99% of all reads have an average latency of 0 ms. Very good results!

On the RamSan-620 fibrechannel interfaces we can see that the IO load is equally spread among all interfaces.


Using ATTO Disk benchmark I tested the throughput by starting a benchmark on one virtual machine. With 8192 KB blocks reads a throughput of 800 MB/sec is reached. In this case the 4-Gigabit HBA of the blade server becomes the bottleneck. The 4-Gigabit HBA delivers 800 MB/sec throughput per channel in full-duplex mode. Since ESX 4 always uses one active path with round-robin path policy, a maximum of 800 MB/sec throughput can be reached.


From these basic performance tests the RamSan-620 SSD storage device proves the be a high performance storage with extremely low latency and is very useful as a tier 0 storage device for a VMware vSphere Infrastructure environment. In the next weeks I will further test the device and post my results.


Adjust Queue Depth on vSphere ESXi

March 23, 2010

The procedure for changing the queue depth of a HBA on vSphere ESXi servers differs a little bit from vSphere ESX servers. For ESXi you can performe this procedure from the vSphere CLI or the vSphere Management Assistant (vMA).

The following example describes the procedure for changing the queue depth for a QLogic adapter:

  1. Verify which QLogic HBA module is currently loaded by entering the following command:esxcfg-module -list | grep qla
  2. Run the following commands. The example shows the qla2xxx module. Use the appropriate module based on the outcome of the previous step.vicfg-module -s ql2xmaxqdepth=64 qla2xxx

    In this case, the HBA represented by qla2xxx will have its LUN queue depth set to 64.

  3. Reboot your host.

HP StorageWorks Enterprise Virtual Array (EVA) best practices

February 26, 2010

HP has released a good document on configuration best practices for HP StorageWorks Enterprise Virtual Array (EVA) family and VMware vSphere 4. It advises in the configuration of your EVA and ESX(i) 4. I hadn’t noticed the article, but since it is a good read, I give you the link anyway:


vSphere, HP EVA and path policies

October 11, 2009

The HP EVA comes in two types of storage arrays. An active/passive array and an active/active array. The HP EVA active/active storage arrays can handle I/O requests on both storage processors. With ESX 3.5 and an active/active EVA array you had two options for the path selection policy, fixed and MRU (Most Recently Used). With an active/active EVA array most people used the fix path policy to manually load balance the I/O over both storage processors. This process is very work intensive (it has to be done on all ESX hosts) and is prone to errors (e.g. a fixed path is configured to the wrong storage processor or path trashing).

With the release of vSphere and the vStorage API’s for multipathing, the round robin path selection policy is now fully supported. The HP EVA active/active storage arrays support ALUA which stands for Asymmetrical Logical Unit Access. ALUA occurs when the access characteristics of one port may differ from those of another port. ALUA allows a lun to be accessed via its primary path (via the owning Storage Processor) and via an asymmetrical path (via the not-owning Storage Processor). I/O to the not-owning Storage Processor or not-optimized path  comes with a performance penalty because the I/O has to be transmitted over the internal connection between the storage processors which does not have much bandwidth.

With ALUA the ESX host is aware of the non-optimized paths. So when you use the round robin path policy, ESX will load balance the I/O over the optimized paths (the paths to the owning storage processor of the lun) and use the non-optimized paths only in case of a failure on the optimized paths. To my opinion this has two advances: both path are used and manually load balancing isn’t necessary anymore which saves a lot of configuration work.

So here you have it! I recommend to use the round robin path policy with an active/active EVA array. If you don’t want to use the round robin path policy I would recommend to use the MRU policy. The MRU policy is ALUA aware and will also use the optimized paths.

vSphere Volume Grow does not work

September 4, 2009

One of the new features of vSphere is the ability to grow a volume/extent. This way you don’t have to add additional extends/lun’s to grow a datastore.

However trying to a grow a volume with vCenter on a ESXi 4 host didn’t work for me. If I clicked on the "Increase" button, a pop-up for Increase DataStore Capacity came up, but it was blank.


vCenter reports 110 GB on the device and 100 GB capacity on the extent. So vCenter did recognize the increase of the lun, but I wasn’t able to use the ability to grow an extent.

After some research I found out that the ability to grow a volume/extent does work on new created datastores on vSphere.

However under some circumstances a datastore upgraded from ESX 3.5 cannot be not be grown from the vSphere client through vCenter. But when you connect directly to the ESX host it can be upgraded! vCenter now presents the datastore as expandable.


Not possible when done via vCenter. How strange?

VMKernel: nmp_DeviceRequestFastDeviceProbe

August 6, 2009

I am currently testing ESXi 4 by adding one ESXi 4 host to a VMware production cluster of a customer. The ESXi 4 host seems to run fine but i noticed the following kernel warnings in the system log:

Jul 18 17:00:27 vmkernel: 2:07:08:24.308 cpu7:40478)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100021b8480) to NMP device "naa.600508b4000554df00007000034a0000" failed on physical path "vmhba1:C0:T0:L11" H:0x2 D:0x0 P:0x0 Possible sense data: Jul 18 17:00:27 0x0 0x0 0x0.

Jul 18 17:00:27 vmkernel: 2:07:08:24.308 cpu7:40478)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508b4000554df00007000034a0000" state in doubt; requested fast path state update…

Jul 18 17:00:27 vmkernel: 2:07:08:24.308 cpu7:40478)ScsiDeviceIO: 747: Command 0x2a to device "naa.600508b4000554df00007000034a0000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

The log message above contains the following codes:

failed H:0x2 D:0x0 P:0x0

The interesting section here is the code starting with "H" (H stands for "Host status"). Host status 0x2 means "HOST BUSY"

Vmware support gives the following explanation for this:

I checked with our bug database and as I had thought previously, H:0x2 D:0x0 P:0x0 translates to hba busy. The driver for whatever reason failed the i/o with a busy status. These can occur for any number of reasons. These failures are automatically retried by ESX.

Jul 18 17:00:27 vmkernel: 2:07:08:24.308 cpu7:40478)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508b4000554df00007000034a0000" state in doubt; requested fast path state update…"

This messaging will initially indicate that a NMP command was not responsive on a device, thus the NMP plugin ‘doubted’ the sate of the lun, i.e was it busy, was it on a path, was it responsive. This could be a driver problem or spurious logging. A bug for this message has been logged, and as yet is not an issue, unless followed by failing I/O or VM failures.

So it looks like a bug, but as yet is not an issue. Hope this gives some clarification!