ESX 6.0でストレージアクセス関連の調査を行った際に使ったesxcliコマンドメモ


esxcli storage core device list

現在認識してるストレージについてのいろんな情報

[root@esxi10:~] esxcli storage core device list
eui.9fa59185cb34c2ab6c9ce9003faf076a
   Display Name: Nimble iSCSI Disk (eui.9fa59185cb34c2ab6c9ce9003faf076a)
   Has Settable Display Name: true
   Size: 5242880
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/eui.9fa59185cb34c2ab6c9ce9003faf076a
   Vendor: Nimble
   Model: Server
   Revision: 1.0
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: supported
   Other UIDs: vml.01000000003966613539313835636233346332616236633963653930303366616630373661536572766572
   Is Shared Clusterwide: true
   Is Local SAS Device: false
   Is SAS: false
   Is USB: false
   Is Boot USB Device: false
   Is Boot Device: false
   Device Max Queue Depth: 128
   No of outstanding IOs with competing worlds: 32
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

[root@esxi10:~]

個別で見る場合は「esxcli storage core device list -d ストレージ指定」

[root@esxi10:~] esxcli storage core device list -d eui.9fa59185cb34c2ab6c9ce9003faf076a
eui.9fa59185cb34c2ab6c9ce9003faf076a
   Display Name: Nimble iSCSI Disk (eui.9fa59185cb34c2ab6c9ce9003faf076a)
   Has Settable Display Name: true
   Size: 5242880
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/eui.9fa59185cb34c2ab6c9ce9003faf076a
   Vendor: Nimble
   Model: Server
   Revision: 1.0
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: supported
   Other UIDs: vml.01000000003966613539313835636233346332616236633963653930303366616630373661536572766572
   Is Shared Clusterwide: true
   Is Local SAS Device: false
   Is SAS: false
   Is USB: false
   Is Boot USB Device: false
   Is Boot Device: false
   Device Max Queue Depth: 128
   No of outstanding IOs with competing worlds: 32
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false
[root@esxi10:~]

ESXiで認識しているファイルシステム一覧

esxcli storage filesystem list

[root@esxi10:~] esxcli storage  filesystem list
Mount Point                                           Volume Name                   UUID                                    Mounted  Type             Size           Free
----------------------------------------------------  ----------------------------  --------------------------------------  -------  ------  -------------  -------------
/vmfs/volumes/5bbc594e-3e173ecc-b3e7-90b11c536183     esxstore-vsphere10            5bbc594e-3e173ecc-b3e7-90b11c536183        true  VMFS-5  5497289703424  2186855579648
/vmfs/volumes/b49358e5-23dc34a5-1c9f-5af965bf7cb0                                   b49358e5-23dc34a5-1c9f-5af965bf7cb0        true  vfat        261853184       79454208
/vmfs/volumes/4aed47ef-d4e7ee55-f394-88684bc4443c                                   4aed47ef-d4e7ee55-f394-88684bc4443c        true  vfat        261853184       79417344
/vmfs/volumes/5d47ebcd-a4552bf1-c3e2-90b11c536183                                   5d47ebcd-a4552bf1-c3e2-90b11c536183        true  vfat        299712512       84172800
[root@esxi10:~]

指定したVMFSの情報を確認

「esxcli storage vmfs host list –volume-label 名前」

[root@esxi10:~] esxcli storage vmfs host list --volume-label esxstore-vsphere10
Heartbeat UUID                       MAC Address        Driver Version  Heartbeat Offset  Locking Mode  Status
-----------------------------------  -----------------  --------------  ----------------  ------------  ------
5d6c6d91-51b61938-424a-90b11c536183  90:b1:1c:53:61:83  5.61                     3178496  ATS-only      Normal
[root@esxi10:~]

ストレージアクセスのQueue関連設定変更

現在の値確認は「esxcli storage core device list -d ストレージ指定」で行う。

「Device Max Queue Depth」を「128」に設定する

[root@esxi10:~]esxcli storage core device set -d eui.9fa59185cb34c2ab6c9ce9003faf076a -m 128
[root@esxi10:~]

「No of outstanding IOs with competing worlds」を「32」に設定する

[root@esxi10:~]esxcli storage core device set -d eui.9fa59185cb34c2ab6c9ce9003faf076a -O 32
[root@esxi10:~]

ストレージパス関連

esxcli storage core path list

[root@esxi10:~] esxcli storage core path list
iqn.1998-01.com.vmware:esxi10-753076d4-00023d000001,iqn.2007-11.com.nimblestorage:esxstore-vsphere10-v371c7edc5d1bd1e3.000000ab.6a07af3f,t,2460-eui.9fa59185cb34c2ab6c9ce9003faf076a
   UID: iqn.1998-01.com.vmware:esxi10-753076d4-00023d000001,iqn.2007-11.com.nimblestorage:esxstore-vsphere10-v371c7edc5d1bd1e3.000000ab.6a07af3f,t,2460-eui.9fa59185cb34c2ab6c9ce9003faf076a
   Runtime Name: vmhba38:C0:T1:L0
   Device: eui.9fa59185cb34c2ab6c9ce9003faf076a
   Device Display Name: Nimble iSCSI Disk (eui.9fa59185cb34c2ab6c9ce9003faf076a)
   Adapter: vmhba38
   Channel: 0
   Target: 1
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi10-753076d4
   Target Identifier: 00023d000001,iqn.2007-11.com.nimblestorage:esxstore-vsphere10-v371c7edc5d1bd1e3.000000ab.6a07af3f,t,2460
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi10-753076d4
   Target Transport Details: IQN=iqn.2007-11.com.nimblestorage:esxstore-vsphere10-v371c7edc5d1bd1e3.000000ab.6a07af3f Alias= Session=00023d000001 PortalTag=2460
   Maximum IO Size: 131072
[root@esxi10:~]

各ストレージのパス選択アルゴリズム確認

「esxcli storage nmp device list」

[root@esxi10:~] esxcli storage nmp device list
eui.9fa59185cb34c2ab6c9ce9003faf076a
   Device Display Name: Nimble iSCSI Disk (eui.9fa59185cb34c2ab6c9ce9003faf076a)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=0,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_MRU
   Path Selection Policy Device Config: Current Path=vmhba38:C0:T1:L0
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba38:C0:T1:L0
   Is USB: false
[root@esxi10:~]

使用できるPSP一覧

[root@esxi10:~] esxcli storage nmp psp list
Name           Description
-------------  ---------------------------------
VMW_PSP_MRU    Most Recently Used Path Selection
VMW_PSP_RR     Round Robin Path Selection
VMW_PSP_FIXED  Fixed Path Selection
[root@esxi10:~]

指定できるSATP一覧

[root@esxi10:~] esxcli storage nmp satp list
Name                 Default PSP    Description
-------------------  -------------  -------------------------------------------------------
VMW_SATP_ALUA        VMW_PSP_MRU    Supports non-specific arrays that use the ALUA protocol
VMW_SATP_MSA         VMW_PSP_MRU    Placeholder (plugin not loaded)
VMW_SATP_DEFAULT_AP  VMW_PSP_MRU    Placeholder (plugin not loaded)
VMW_SATP_SVC         VMW_PSP_FIXED  Placeholder (plugin not loaded)
VMW_SATP_EQL         VMW_PSP_FIXED  Placeholder (plugin not loaded)
VMW_SATP_INV         VMW_PSP_FIXED  Placeholder (plugin not loaded)
VMW_SATP_EVA         VMW_PSP_FIXED  Placeholder (plugin not loaded)
VMW_SATP_ALUA_CX     VMW_PSP_RR     Placeholder (plugin not loaded)
VMW_SATP_SYMM        VMW_PSP_RR     Placeholder (plugin not loaded)
VMW_SATP_CX          VMW_PSP_MRU    Placeholder (plugin not loaded)
VMW_SATP_LSI         VMW_PSP_MRU    Placeholder (plugin not loaded)
VMW_SATP_DEFAULT_AA  VMW_PSP_FIXED  Supports non-specific active/active arrays
VMW_SATP_LOCAL       VMW_PSP_FIXED  Supports direct attached devices
[root@esxi10:~]

パス選択アルゴリズムの変更

[root@esxi10:~] esxcli storage nmp device set --psp=VMW_PSP_RR --device=eui.9fa59185cb34c2ab6c9ce9003faf076a
[root@esxi10:~]

変更確認

[root@esxi10:~] esxcli storage nmp device list -d eui.9fa59185cb34c2ab6c9ce9003faf076a
eui.9fa59185cb34c2ab6c9ce9003faf076a
   Device Display Name: Nimble iSCSI Disk (eui.9fa59185cb34c2ab6c9ce9003faf076a)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=0,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba38:C3:T1:L0, vmhba38:C0:T1:L0
   Is USB: false
[root@esxi10:~]

HBAドライバパラメータ調整

インストールされているドライババージョン等を確認

[root@esxi10:~] esxcli software vib list|grep qla
ima-qla4xxx                    2.02.18-1vmw.600.0.0.2494585           VMware  VMwareCertified   2019-08-05
scsi-qla4xxx                   5.01.03.2-7vmw.600.0.0.2494585         VMware  VMwareCertified   2019-08-05
[root@esxi10:~]

HBAで指定できるパラメータ確認

「esxcli system module parameters list -m モジュール名」

[root@esxi10:~] esxcli system module parameters list -m qlnativefc
Name                        Type  Value  Description
--------------------------  ----  -----  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ql2name                     int          Option to have different node and port name for ISP82xxpass value based on the port used on the FCoE switch.
ql2xallocfwdump             int          Option to enable allocation of memory for a firmware dump during HBA initialization.  Memory allocation requirements vary by ISP type.  Default is 1 - allocate memory.
ql2xattemptdumponpanic      int          Attempt fw dump for each function on PSOD Default is 0 - Don't attempt fw dump.
ql2xbypass_log_throttle     int          Option to bypass log throttling.Default is 0 - Throttling enabled. 1 - Log all errors.
ql2xcmdtimeout              int          Timeout value in seconds for scsi command, default is 20
ql2xcmdtimermin             int          Minimum command timeout value. Default is 30 seconds.
ql2xdbwr                    int          Option to specify scheme for request queue posting. 0 -- Regular doorbell. 1 -- (Default) CAMRAM doorbell (faster).
ql2xdevdiscgoldfw           int          Option to enable device discovery with golden firmware Default is 0 - no discovery. 1 - discover device.
ql2xdisablenpiv             int          Option to disable/enable NPIV feature globally.        1 - NPIV disabled. Default is 0 - NPIV enabled.
ql2xdontresethba            int          1: Do not reset on failure, 0: Reset on failure, default is 0
ql2xdontresethba_83xx       int          1: Do not reset on failure, 0: Reset on failure, default is 0
ql2xenablehba_err_chk       int           Enable T10-CRC-DIF Error isolation by HBA:
 Default is 2.
  0 -- Error isolation disabled
  1 -- Error isolation enabled only for DIX Type 0
  2 -- Error isolation enabled for all Types
ql2xenablemd                int          82XX/8044: Option to enable/disable the minidump capture capability.     0 - disable minidump capture          1 - (Default) enable minidump capture
ql2xenablemsi2422           int          Enables MSI interrupt scheme on 2422sDefault is 0 - disable MSI-X/MSI. 1 - enable MSI-X/MSI.
ql2xenablemsi24xx           int          Enables MSIx/MSI interrupt scheme on 24xx cardsDefault is 1 - enable MSI-X/MSI. 0 - disable MSI-X/MSI.
ql2xenablemsix              int          Set to enable MSI or MSI-X interrupt mechanism. 0 = enable traditional pin-based interrupt mechanism. 1 = enable MSI-X interrupt mechanism (Default). 2 = enable MSI interrupt mechanism.
ql2xenablesmartsan          int          Enable HP Smart SAN management feature.Default is 1.
 0 - Disable the Smart SAN feature.  1 - Enable the Smart SAN feature.
ql2xexecution_throttle      int          IOCB exchange count for HBA.Default is 0, set intended value to override Firmware defaults.
ql2xextended_error_logging  int          Option to enable extended error logging, Default is 0 - no logging. 1 - log errors.
ql2xfabricpriorityqos       int           Option to set QoS mode
 Default is 2.
 0 - Fabric Priority QoS is disabled
 1 - Queue-based Fabric Priority QoS enabled.
 2 - IOCB-based Fabric Priority QoS enabled.
ql2xfdmienable              int          Enables FDMI registratons Default is 1 - perfom FDMI. 0 - no FDMI.
ql2xfwloadbin               int          Option to specify location from which to load ISP firmware: 2 -- load firmware via the requestFirmware() (hotplug)      interface. 1 -- load firmware from flash. 0 -- use default semantics.
ql2xiidmaenable             int          Enables iIDMA settings Default is 1 - perform iIDMA. 0 - no iIDMA.
ql2xintrdelaytimer          int          ZIO: Waiting time for Firmware before it generates an interrupt to the host to notify completion of request.
ql2xioctltimeout            int          IOCTL timeout value in seconds for pass-thur commands. Default is 66 seconds.
ql2xioctltimertest          int          IOCTL timer test enable - set to enable ioctlcommand timeout value to trigger before fw cmdtimeout value. Default is disabled
ql2xloadfwbin               int          Option to Load Firmware from the flash/driver source for qla82xx Default is 1, 0 = load firmware bundled with driver source 1 = load from Flash
ql2xloginretrycount         int          Specify an alternate value for the NVRAM login retry count.
ql2xlogintimeout            int          Login timeout value in seconds.
ql2xmaxlun                  int          Defines the maximum LUNs to register with the SCSI midlayer.    Default is 256. Maximum allowed is 65535.
ql2xmaxqdepth               int          Maximum queue depth to report for target devices.
ql2xmaxsgs                  int          Maximum scatter/gather entries per request,Default is the Max the OS Supports.
ql2xmdcapmask               int          82XX/8044: Option to set the driver capture mask for f/w minidump as follows:          0x00 - use f/w recommended capture mask         0x03 - capture mask 0x3         0x0F - capture mask 0x0F      0x1F - (Default) capture mask 0x1F      0x7F - capture mask 0x7F
ql2xmqcpuaffinity           int          Enables CPU affinity settings for the driver Default is 1 to turn on the cpu affinity.Set to 0 for no affinity of request and response IO.
ql2xmqqos                   int          Enables QoS settings Default is 1. Set it to enable            queues in MQ QoS mode.
ql2xoperationmode           int          Option to disable ZIO mode for ISP24XX:   Default is 1, set 0 to disable
ql2xplogiabsentdevice       int          Option to enable PLOGI to devices that are not present after a Fabric scan.  This is needed for several broken switches. Default is 0 - no PLOGI. 1 - perfom PLOGI.
ql2xshiftctondsd            int          Set to control shifting of command type processing based on total number of DSD.
ql2xt10difvendor            int           Vendor unique T10 DIF support.  Default is enabled.
 0 -- Disabled.
 1 -- Enabled.
ql2xt10enable               int           Enable T10-CRC-DIF  Default is 1 - DIF Support Enabled. 0 - Disable it
ql2xt10guard                int           Adapter guard type supported  Default is CRC checksumming and IP checksumming.
ql2xt10protmask             int           Adapter protection mask  Default is 0x55 - T10 PI Type 1/3, DIX Type 1/3 protection.
ql2xusedefmaxrdreq          int          Default is 0 - adjust PCIe Maximum Read Request Size. 1 - use system default.
ql2xuseshadowregisters      int           For ISP27XX/ISP28XX : Option to use shadow register for request/response queues :
 Default is 0.
  0 -- Don't use shadow registers for the request/response queue manipulation
  1 -- Use shadow registers for the request/response queue manipulation
ql2xvmidexpiration          int           Expiration time for VMID in minutes. 0 is no expiration. Default is 240 minutes (4 hours) with no I/O leads to de-registration.
ql2xvmidsupport             int           Enable VMID support in driver.  Default is disabled.
 0 -- Disabled.
 1 -- Enabled.
ql2xvmidtgtind              int           Ignore target settings for QoS VMID.  Default is disabled.
 0 -- Disabled.
 1 -- Enabled.
qlport_down_retry           int          Maximum number of command retries to a port that returns a PORT-DOWN status.
[root@esxi10:~]

「esxcfg-module –show-info モジュール名」

[root@esxi10:~] esxcfg-module --show-info qlnativefc
esxcfg-module module information
 input file: /usr/lib/vmware/vmkmod/qlnativefc
 License: BSD
Cannot access module instance.
[root@esxi10:~]

値の設定

[root@esxi10:~] esxcfg-module --get-options qlnativefc
qlnativefc enabled = 1 options = ''
[root@esxi10:~]

コメントを残す

メールアドレスが公開されることはありません。 * が付いている欄は必須項目です

このサイトはスパムを低減するために Akismet を使っています。コメントデータの処理方法の詳細はこちらをご覧ください