How To Check HBA Queue Depth In ESXi?

Checking HBA queue depth in ESXi is a critical task for diagnosing storage performance issues, as a full or overloaded queue is a common cause of latency. The primary method is to use the esxtop command-line utility in the ESXi Shell. This tool provides real-time statistics on storage queues at different levels: the host bus adapter (HBA) itself, the storage devices (LUNs), and the individual virtual machines.

Understanding ESXi storage queues

Before diving into the commands, it's important to understand the different queue levels in ESXi's storage stack. This will help you interpret the esxtop output correctly and pinpoint the source of performance problems.

Host Bus Adapter (HBA) Queue: This is the queue managed by the HBA driver (e.g., QLogic, Emulex). It holds I/O commands sent from the ESXi kernel to the physical adapter. If this queue is full, the host cannot submit any more I/O requests, leading to increased latency.
Device Queue: This is the queue for a specific storage device, or LUN. It's the number of I/O requests that a particular LUN can handle from the ESXi kernel. The device queue depth (DQLEN) is usually limited by the HBA queue depth or a per-device setting (Disk.SchedNumReqOutstanding).
Virtual Machine (VM) Queue: The virtual SCSI controller within a VM also has a queue. If this queue is full, the VM will experience high latency, but it won't be reflected at the ESXi or storage array level.

Using `esxtop` to check queue depth

The esxtop utility is your primary tool for monitoring storage queues in real-time.

Access the ESXi Shell. Enable SSH on your ESXi host from the vSphere Client and connect using an SSH client.
**Run esxtop**sh
```
esxtop
```
Use code with caution.
View disk device statistics. Once esxtop is running, press d to switch to the disk device view.
Add queue statistics. To see the queue depth columns, press f to manage the field display. In the menu, select c to add the "Queue Stats" group, then press Enter.
Interpret the output. The key columns for queue depth analysis are:
- AQLEN: The active queue length. This is the maximum number of active I/O commands the HBA driver is configured to support.
- DQLEN: The device queue length. This is the maximum number of active commands a specific storage device (LUN) can support.
- QUED: The number of I/O commands waiting in the queue to be sent to the storage device. A consistently high QUED value indicates a bottleneck.
- ACTV: The number of active I/O commands currently being processed.
- %USD: The percentage of the queue depth being used.

Pro Tip: In the esxtop disk view, press u to see per-device LUN statistics, which can help narrow down issues to a specific storage target.

Using `esxcli` for static queue depth configuration

While esxtop is for real-time monitoring, esxcli is used to see and modify the configured queue depth parameters. This is especially useful for checking the host-wide and per-device settings.

List storage devices. To get the NAA ID of your storage devices, run the command:sh
```
esxcli storage core device list
```
Use code with caution.
Check HBA driver parameters. HBA queue depth is often a module parameter. You first need to find the correct driver name (e.g., qlnativefc, lpfc).sh
```
esxcli system module list | grep <driver_name>
```
Use code with caution.
List module parameters. Use the driver name to check its parameters, which include the queue depth setting.sh
```
esxcli system module parameters list -m <driver_name>
```
Use code with caution.

For example, for QLogic HBAs, the output might include ql2xmaxqdepth.
Check device-specific queue settings. For a specific LUN (replace naa.xxxx with your device ID), check its configuration, which includes the number of outstanding I/Os.sh
```
esxcli storage core device list -d <naa.xxxx>
```
Use code with caution.

Understanding and addressing queue depth bottlenecks

A storage queue bottleneck occurs when the number of I/O requests from the virtual machines exceeds the maximum capacity of the HBA or storage device. This causes requests to queue up (QUED > 0 in esxtop), leading to increased latency.

Causes of queue depth bottlenecks

Storage array overload: The storage processor or front-end ports are saturated, and they cannot handle the I/O load.
Inadequate HBA queue depth: The HBA's configured queue depth is too low for the workload it's handling.
Misconfigured device queue depth: The Disk.SchedNumReqOutstanding value for a LUN may be set too low.
Improper workload distribution: A single, high-intensity VM is monopolizing a LUN or HBA, starving other VMs of I/O resources.

Best practices and resolution

Identify the bottleneck: Use esxtop to determine where the queue is building up. If QUED is consistently high and DQLEN is maxed out, the device queue is the bottleneck. If AQLEN is the issue, the HBA is the problem.
Increase HBA queue depth: If your HBA queue is the bottleneck, consult your hardware and storage vendor for recommended settings. Increasing the value with esxcli system module parameters set may improve performance, but it requires a host reboot.
Increase per-device queue depth: For LUNs, you can increase the number of outstanding I/Os using esxcli storage core device set -d <naa.xxxx> -O <value>.
Distribute workloads: If one VM is causing the bottleneck, consider moving it to a less utilized datastore or using multiple datastores across different LUNs to distribute the I/O load.
Hardware and firmware checks: Ensure your HBAs have the latest drivers and firmware, as these often contain performance enhancements.
Consult with storage vendor: Always check with your storage array vendor and VMware for guidance before making changes, as increasing queue depths can sometimes lead to array-level saturation.

Enjoyed this article? Share it with a friend.