samba 4.23.3 で立てたActive Directoryサーバの機能レベルが2008R2から動かせない件を修正する

ESXi8 Free環境上に Active Directoryサーバを立てるか、と、AlmaLinux 9 で samba 4.23.3 をソースからコンパイルして構築した

# /usr/local/samba/bin/samba-tool domain provision --use-rfc2307 --interactive
Realm [ADSAMPLE.LOCAL]:
Domain [ADSAMPLE]:
Server Role (dc, member, standalone) [dc]:
DNS backend (SAMBA_INTERNAL, BIND9_FLATFILE, BIND9_DLZ, NONE) [SAMBA_INTERNAL]:
DNS forwarder IP address (write 'none' to disable forwarding) [8.8.8.8]:  8.8.8.8
Administrator password:
Retype password:
INFO 2025-11-10 14:24:37,370 pid:1551 /usr/local/samba/lib64/python3.9/site-packages/samba/provision/__init__.py #2112: Looking up IPv4 addresses
<略>
INFO 2025-11-10 14:24:49,826 pid:1551 /usr/local/samba/lib64/python3.9/site-packages/samba/provision/__init__.py #501: DOMAIN SID:            S-1-5-21-1830428519-1651848948-1698044471
#

これで起動したActive Directoryサーバのフォレストレベル / ドメインレベル は下記の様にWindows 2008 R2 となっていた。

# samba-tool domain level show
Domain and forest function level for domain 'DC=adsample,DC=local'

Forest function level: (Windows) 2008 R2
Domain function level: (Windows) 2008 R2
Lowest function level of a DC: (Windows) 2008 R2
#

これをアップグレードしようと samba-tool domain level raiseコマンドを実行してみてもエラーとなる。

# samba-tool domain level raise --forest-level=2012_R2
ERROR: Forest function level can't be higher than the domain function level(s). Please raise it/them first!
# samba-tool domain level raise --domain-level=2012_R2
ERROR: Domain function level can't be higher than the lowest function level of a DC!
#

これはデフォルトのsamba設定で”ad dc functional level”が2008R2までとなっているからそういうことになっているのだという(参考:Samba domain controller: raising (all kinds of) level)

testparamコマンドを実行して現在の設定値を確認する

# /usr/local/samba/bin/testparm -s --section-name=global --parameter-name="ad dc functional level"
Load smb config files from /usr/local/samba/etc/smb.conf
Loaded services file OK.
Weak crypto is allowed by GnuTLS (e.g. NTLM as a compatibility fallback)

2008_R2
#

現状の /usr/local/samba/etc/smb.conf に記載はないが、 samba設定としては 2008_R2 として認識されている、ということを確認出来た

この結果を受けて/usr/local/samba/etc/smb.conf のglobalセクションに「ad dc functional level = 2016」という記述を追加する

# cat /usr/local/samba/etc/smb.conf
# Global parameters
[global]
        dns forwarder = 8.8.8.8
        netbios name = ADSERVER
        realm = ADSAMPLE.LOCAL
        server role = active directory domain controller
        workgroup = ADSAMPLE
        idmap_ldb:use rfc2307 = yes
        ad dc functional level = 2016

[sysvol]
        path = /usr/local/samba/var/locks/sysvol
        read only = No

[netlogon]
        path = /usr/local/samba/var/locks/sysvol/adsample.local/scripts
        read only = No
#

testparamで記述が反映されているかを確認

# /usr/local/samba/bin/testparm -s --section-name=global --parameter-name="ad dc functional level"
Load smb config files from /usr/local/samba/etc/smb.conf
Loaded services file OK.
Weak crypto is allowed by GnuTLS (e.g. NTLM as a compatibility fallback)

2016
#

sambaを再起動して、機能レベルがどうなったのかを確認

# systemctl restart samba-ad-dc
# samba-tool domain level show
Domain and forest function level for domain 'DC=adsample,DC=local'

Forest function level: (Windows) 2008 R2
Domain function level: (Windows) 2008 R2
Lowest function level of a DC: (Windows) 2016
#

Lowest function level of a DC が変更されたので、上2つも変更できるようになった

まずはドメインの機能レベルを変更

# samba-tool domain level raise --domain-level=2012_R2
Domain function level changed!
All changes applied successfully!
# samba-tool domain level show
Domain and forest function level for domain 'DC=adsample,DC=local'

Forest function level: (Windows) 2008 R2
Domain function level: (Windows) 2012 R2
Lowest function level of a DC: (Windows) 2016
#

続いてフォレストの機能レベルを変更

# samba-tool domain level raise --forest-level=2012_R2
Forest function level changed!
All changes applied successfully!
# samba-tool domain level show
Domain and forest function level for domain 'DC=adsample,DC=local'

Forest function level: (Windows) 2012 R2
Domain function level: (Windows) 2012 R2
Lowest function level of a DC: (Windows) 2016
#

これで問題なくなった。

iscsiadmコマンドのメモ

HPE VM Essentails に iSCSIストレージをつないだ場合の動作がわからない点が多かった、Web UIからではなく、CLIでいろいろ調べる羽目になったのでメモ書き

Linux汎用で使える話ではある

接続の確認

iscsiが接続できているかを「iscsiadm -m session」で確認

pcuser@hpevme6:~$ sudo iscsiadm -m session
tcp: [1] 192.168.3.34:3260,1029 iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3 (non-flash)
tcp: [2] 192.168.2.34:3260,1028 iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3 (non-flash)
pcuser@hpevme6:~$

何もつながっていない場合は下記

pcuser@hpevme6:~$ sudo iscsiadm -m session
iscsiadm: No active sessions.
pcuser@hpevme6:~$

詳細を確認したい場合は「-P 数字」というオプションを付ける。0,1,2,3が指定できるが「-P 0」は付けない場合と同じ

0~2は、接続先IPアドレスとログイン情報などの範囲
3になると、デバイスが認識されているかがわかるようになるので「sudo iscsiadm -m session -P 3」はトラブル時に必須

pcuser@hpevme6:~$ sudo iscsiadm -m session --print=3
iSCSI Transport Class version 2.0-870
version 2.1.9
Target: iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3 (non-flash)
        Current Portal: 192.168.3.34:3260,1029
        Persistent Portal: 192.168.3.34:3260,1029
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.3.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 33 State: running
                scsi33 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdb          State: running
                scsi33 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdd          State: running
        Current Portal: 192.168.2.34:3260,1028
        Persistent Portal: 192.168.2.34:3260,1028
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.2.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 34 State: running
                scsi34 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdc          State: running
                scsi34 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sde          State: running
pcuser@hpevme6:~$

“Attached SCSI devices:” のあとに scsi~ という表記があるかどうか

ない場合は、iSCSIストレージ側で、アクセス許可されてない可能性があるので、設定を確認

まず、Linux側のInitiatorNameを確認。Linuxの場合 /etc/iscsi/initiatorname.iscsi に記載されいて、OSインストール直後などは「InitiatorName=iqn.2004-10.com.ubuntu:01:<ランダム>」といった値で設定されていることが多い

HPE VMEの場合、hpe-vmセットアップ直後は ubuntuランダムなのだが、Web UIからiSCSI接続をするとホスト名 ランダムといった下記のような設定に切り替わる

pcuser@hpevme6:~$ sudo cat /etc/iscsi/initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.2024-12.com.hpe:hpevme6:59012
pcuser@hpevme6:~$

この「InitiatorName」の値をiSCSIストレージ側の「イニシエータ」の登録に追加する必要がある

NetAppの場合の設定例

HPE VMEの場合、iSCSI設定を行う際に、Manager仮想マシンが各サーバの /etc/iscsi/initiatorname.iscsi の値を書き換えるので、設定したはずなのにつながらない場合は、最新の名前がiSCSIストレージ側に登録されているかを確認すること

設定変更した後、「sudo iscsiadm -m session –rescan」を実行して再スキャンを行う

認識していない状態から–rescanを実行して認識した、という実行ログ

pcuser@hpevme6:~$ sudo iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.1.9
Target: iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3 (non-flash)
        Current Portal: 192.168.3.34:3260,1029
        Persistent Portal: 192.168.3.34:3260,1029
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.3.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 33 State: running
        Current Portal: 192.168.2.34:3260,1028
        Persistent Portal: 192.168.2.34:3260,1028
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.2.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 34 State: running
pcuser@hpevme6:~$ sudo iscsiadm -m session --rescan
Rescanning session [sid: 1, target: iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3, portal: 192.168.3.34,3260]
Rescanning session [sid: 2, target: iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3, portal: 192.168.2.34,3260]
pcuser@hpevme6:~$ sudo iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.1.9
Target: iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3 (non-flash)
        Current Portal: 192.168.3.34:3260,1029
        Persistent Portal: 192.168.3.34:3260,1029
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.3.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 33 State: running
                scsi33 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdb          State: running
                scsi33 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdd          State: running
        Current Portal: 192.168.2.34:3260,1028
        Persistent Portal: 192.168.2.34:3260,1028
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.2.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 34 State: running
                scsi34 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdc          State: running
                scsi34 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sde          State: running
pcuser@hpevme6:~$

マルチパスの認識

iSCSIストレージは複数のセッション=マルチパスで接続されるので、下の例では、scsi33とscsi34 の2つで見えている

pcuser@hpevme6:~$ sudo iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.1.9
Target: iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3 (non-flash)
        Current Portal: 192.168.3.34:3260,1029
        Persistent Portal: 192.168.3.34:3260,1029
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.3.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 33 State: running
                scsi33 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdb          State: running
                scsi33 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdd          State: running
        Current Portal: 192.168.2.34:3260,1028
        Persistent Portal: 192.168.2.34:3260,1028
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2024-12.com.hpe:hpevme6:59012
                Iface IPaddress: 192.168.2.60
                Iface HWaddress: default
                Iface Netdev: default
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 1048576
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 34 State: running
                scsi34 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdc          State: running
                scsi34 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sde          State: running
pcuser@hpevme6:~$

2パスで見えているものを1つにまとめるのが multipathd の役割

「sudo multipath -ll」を実行して認識状況を確認

pcuser@hpevme6:~$ sudo multipath -ll
3600a09807770457a795d5a4159416c34 dm-2 NETAPP,LUN C-Mode
size=70G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 34:0:0:0 sdc 8:32 active ready running
  `- 33:0:0:0 sdb 8:16 active ready running
3600a09807770457a795d5a4159416c35 dm-1 NETAPP,LUN C-Mode
size=5.0G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 33:0:0:1 sdd 8:48 active ready running
  `- 34:0:0:1 sde 8:64 active ready running
pcuser@hpevme6:~$

multipathdでまとめられたデバイスは /dev/mapper の下にデバイスファイルがある

pcuser@hpevme6:~$ ls /dev/mapper/*
/dev/mapper/3600a09807770457a795d5a4159416c34  /dev/mapper/control
/dev/mapper/3600a09807770457a795d5a4159416c35  /dev/mapper/ubuntu--vg-ubuntu--lv
pcuser@hpevme6:~$

「sudo multipath -ll」で何も表示されていない場合は、手動でデバイスを登録する

まず、認識している /dev/sd? に対応するWWIDを調べるため「/lib/udev/scsi_id -g -u -d /dev/sd?」を実行する

/lib/udev/scsi_id -g -u -d /dev/sdX

このWWIDをmutlipathに登録するため「multipath -a WWID」を実行する

multipath -a WWID

登録した後は「multipath -r」で再読み込みして、「multipath -ll」で追加されたかを確認する

ターゲットログインなどの初期設定

「iscsiadm -m discovery -t sendtargets -p IPアドレス」で接続

接続パラメータの変更

現在のパラメータ確認は「sudo iscsiadm -m node」でポータル名を確認

pcuser@hpevme6:~$ sudo iscsiadm -m node
192.168.2.34:3260,1028 iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3
192.168.3.34:3260,1029 iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3
pcuser@hpevme6:~$

各ポータルに設定されているパラメータを「sudo iscsiadm -m node -p <ポータル名>」で確認

pcuser@hpevme6:~$ sudo iscsiadm -m node -p 192.168.2.34:3260,1028
# BEGIN RECORD 2.1.9
node.name = iqn.1992-08.com.netapp:sn.e56cfbb6bab111f09b2a000c2980b7f5:vs.3
node.tpgt = 1028
node.startup = automatic
node.leading_login = No
iface.iscsi_ifacename = default
iface.net_ifacename = <empty>
iface.ipaddress = <empty>
iface.prefix_len = 0
iface.hwaddress = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
iface.state = <empty>
iface.vlan_id = 0
iface.vlan_priority = 0
iface.vlan_state = <empty>
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.bootproto = <empty>
iface.subnet_mask = <empty>
iface.gateway = <empty>
iface.dhcp_alt_client_id_state = <empty>
iface.dhcp_alt_client_id = <empty>
iface.dhcp_dns = <empty>
iface.dhcp_learn_iqn = <empty>
iface.dhcp_req_vendor_id_state = <empty>
iface.dhcp_vendor_id_state = <empty>
iface.dhcp_vendor_id = <empty>
iface.dhcp_slp_da = <empty>
iface.fragmentation = <empty>
iface.gratuitous_arp = <empty>
iface.incoming_forwarding = <empty>
iface.tos_state = <empty>
iface.tos = 0
iface.ttl = 0
iface.delayed_ack = <empty>
iface.tcp_nagle = <empty>
iface.tcp_wsf_state = <empty>
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.tcp_timestamp = <empty>
iface.redirect = <empty>
iface.def_task_mgmt_timeout = 0
iface.header_digest = <empty>
iface.data_digest = <empty>
iface.immediate_data = <empty>
iface.initial_r2t = <empty>
iface.data_seq_inorder = <empty>
iface.data_pdu_inorder = <empty>
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
iface.chap_auth = <empty>
iface.bidi_chap = <empty>
iface.strict_login_compliance = <empty>
iface.discovery_auth = <empty>
iface.discovery_logout = <empty>
node.discovery_address = 192.168.2.34
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = 0
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.nr_sessions = 1
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.auth.chap_algs = MD5
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.session.scan = auto
node.session.reopen_max = 0
node.conn[0].address = 192.168.2.34
node.conn[0].port = 3260
node.conn[0].startup = automatic
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD
pcuser@hpevme6:~$

マルチパスで一部のセッションが切れた時の再接続にかかる時間がnode.session.timeo.replacement_timeout で設定されていれ標準は120秒となっている

これだと長いので、例えばHPEの「HPE Primera Red Hat Enterprise Linux実装ガイド」では 10秒 としている

今すぐ変更したい場合はiscsiadmを実行

pcuser@hpevme6:~$ sudo iscsiadm -m node -p 192.168.2.34:3260,1028 |grep node.session.timeo.replacem
ent_timeout
node.session.timeo.replacement_timeout = 120
pcuser@hpevme6:~$ sudo iscsiadm -m node -p 192.168.2.34:3260,1028 -o update -n node.session.timeo.replacement_timeout -v 10
pcuser@hpevme6:~$ sudo iscsiadm -m node -p 192.168.2.34:3260,1028 |grep node.session.timeo.replacement_timeout
node.session.timeo.replacement_timeout = 10
pcuser@hpevme6:~$

恒久的に変更するには /etc/iscsi/iscsid.conf にて該当する行を修正する。

HPE Morpheus VM EssentialsのHCI構成を組んでみた

vSphere代替とも言われるHPE Morpheus VM Essentials (HPE VME, HVM, HPE VM Essentails) で、クラスタをセットアップする際に下記の選択肢がある。

「HVM 1.2 HCI Ceph Cluster on HVM/Ubuntu 24.04」ということで、共有ストレージとしてCephを使用するHCI構成があるらしい。

どういう構成を組めばいいのかわからなかったのですが、ドキュメントを探すと [Infrastructure]-[Clusters]-[HVM Clusters]-[Base Cluster Details]という非常にわかりづらいところに、HCI構成の場合に求める仕様が書いてあった。

・物理サーバが最低3台
・CPUコア数1以上
・メモリ4GB以上。Cephを使う場合ディスク1個ごとに4GB追加
・OSディスク 20GB以上、Ceph用データディスク500GB以上

Example Cluster Deployment」にサンプル構成がある

まずは、最低限のスペックでUbuntu 24.04+HVMをインストールして、Manager仮想マシンをセットアップした。

クラスタを作るところまでの手順は省略

で・・・私がはまった点の1つとして、レイアウト選択のバグ動作、というのがあります。

ver 8.0.10で実施したところ、「HVM 1.2 HCI Ceph Cluster on HVM/Ubuntu 24.04」を選択したところ下記の様にSSHホストが1つしか選択できない状態でした。

そういうものかと思って手順を進めて作成を開始すると指定してないサーバが2つ登場してプロセスが開始されるという状況に・・・

何度かクラスタレイアウトの選択をやり直すと、下記の様に3サーバ分の空欄が表示されるときがありました。この状態であればHCIクラスタの作成に成功しました。

下記の様に入力し作成を開始

作成完了

まあ、わかってしまえば構成自体は簡単だったのですが・・・

Cephのステータスを確認する画面が無いのはどうかと思うんですよ・・・

たとえば[インフラストラクチャ]-[クラスター]で作成したHCIクラスタを表示すると下記の様に「CEPH」って表示があります。

問題は「ステータス:WARN」って表示以上のことを調べるインタフェースが無い、ということ

[ストレージ]で確認できるものは下記の情報だけで、Cephの状態について確認できない

ESXi 8.0でNVMe SSDをUSBケースでつないでデータ移行したら面倒くさいことになった

先日セットアップしたミニPCにESXi8.0 Freeでは余ってたM.2 SATA 256GBにシステムを、M.2 NVMe 512GBを主データストアとして使っていた。

ふと手持ちのM.2系ストレージを見てみると、M.2 NVMeの2TB SSDが2枚余っていたので、片方をESXi用とするか、とまずはUSB NVMeケースに入れてVMFSでフォーマットし、M.2 NVMe 512GB からデータを移動させた。

ちなみに、USB NVMeケースを認識させるには、ESXi8.0で標準動作しているUSB パススルー用の USB Arbitrator service を停止させる必要があった。

出典:Configuring a vSphere ESXi host to use a local USB device for VMkernel coredumps

# /etc/init.d/usbarbitrator stop

M.2 NVMe 2TB を内蔵させてESXi Host Clientから確認

[ストレージ]-[デバイス]ではちゃんとSPD SP7002D2TNGH が認識されている

クリックすると、中にVMFSパーテーションがあるのも認識されている

しかし[ストレージ]-[データストア]には表示されていない。

どういうことなのか、いろいろ調べた結果、なんとか解決した

どうやら、USB NVMeケースでマウントしていたVMFS領域について、明示的にumountしておかないと、いろんな処理がらみで面倒なことになっていたのではないか、と推測される状態となっていた。

まず、M.2 NVMeストレージとして認識されているかを「esxcli nvme namespace list」と「esxcli nvme controller list」を実行して確認

[root@esxi:~] esxcli nvme namespace list
Name                                                                   Controller Number  Namespace ID  Block Size  Capacity in MB
---------------------------------------------------------------------  -----------------  ------------  ----------  --------------
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000                256             1         512         1953514
[root@esxi:~] esxcli nvme controller list
Name                                                                                      Controller Number  Adapter  Transport Type  Is Online  Controller Type  Is VVOL  Keep Alive Timeout  IO Queue Number  IO Queue Size
----------------------------------------------------------------------------------------  -----------------  -------  --------------  ---------  ---------------  -------  ------------------  ---------------  -------------
nqn.2014-08.org.nvmexpress_1e4b_SPD_SP700-2TNGH_________________________0901SP7007D00399                256  vmhba1   PCIe                 true                     false                   0                1           1024
[root@esxi:~]

次に /vmfs/devices/disks/ 以下にデバイスがあるかを確認

[root@esxi:~] ls /vmfs/devices/disks/
t10.ATA_____W800S_256GB_____________________________2202211088199_______
t10.ATA_____W800S_256GB_____________________________2202211088199_______:1
t10.ATA_____W800S_256GB_____________________________2202211088199_______:5
t10.ATA_____W800S_256GB_____________________________2202211088199_______:6
t10.ATA_____W800S_256GB_____________________________2202211088199_______:7
t10.ATA_____W800S_256GB_____________________________2202211088199_______:8
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000:1
vml.0100000000303230305f303030305f303030305f3030303000535044205350
vml.0100000000303230305f303030305f303030305f3030303000535044205350:1
vml.01000000003232303232313130383831393920202020202020573830305320
vml.01000000003232303232313130383831393920202020202020573830305320:1
vml.01000000003232303232313130383831393920202020202020573830305320:5
vml.01000000003232303232313130383831393920202020202020573830305320:6
vml.01000000003232303232313130383831393920202020202020573830305320:7
vml.01000000003232303232313130383831393920202020202020573830305320:8
vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2
vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2:1
[root@esxi:~]

今回認識していないのはSPDのDevfs pathを「esxcli storage core device list」で確認

[root@esxi:~] esxcli storage core device list
t10.ATA_____W800S_256GB_____________________________2202211088199_______
   Display Name: Local ATA Disk (t10.ATA_____W800S_256GB_____________________________2202211088199_______)
   Has Settable Display Name: true
   Size: 244198
   Device Type: Direct-Access
   Multipath Plugin: HPP
   Devfs Path: /vmfs/devices/disks/t10.ATA_____W800S_256GB_____________________________2202211088199_______
   Vendor: ATA
   Model: W800S 256GB
   Revision: 3G5A
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unsupported
   Other UIDs: vml.01000000003232303232313130383831393920202020202020573830305320
   Is Shared Clusterwide: false
   Is SAS: false
   Is USB: false
   Is Boot Device: true
   Device Max Queue Depth: 31
   No of outstanding IOs with competing worlds: 31
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   Display Name: Local NVMe Disk (t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000)
   Has Settable Display Name: true
   Size: 1953514
   Device Type: Direct-Access
   Multipath Plugin: HPP
   Devfs Path: /vmfs/devices/disks/t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   Vendor: NVMe
   Model: SPD SP700-2TNGH
   Revision: SP02203A
   SCSI Level: 0
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: no
   Attached Filters:
   VAAI Status: unsupported
   Other UIDs: vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2
   Is Shared Clusterwide: false
   Is SAS: false
   Is USB: false
   Is Boot Device: false
   Device Max Queue Depth: 1023
   No of outstanding IOs with competing worlds: 32
   Drive Type: physical
   RAID Level: NA
   Number of Physical Drives: 1
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false
[root@esxi:~]

パーテーションは1番の方なので下記を実施

[root@esxi:~] voma -m vmfs -f check -N -d /vmfs/devices/disks/vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2:1
Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done

Checking for filesystem activity
Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
         Reservation Support is not present for NVME devices
Performing filesystem liveness check..|
########################################################################
#   Warning !!!                                                        #
#                                                                      #
#   You are about to execute VOMA without device reservation.          #
#   Any access to this device from other hosts when VOMA is running    #
#   can cause severe data corruption                                   #
#                                                                      #
#   This mode is supported only under VMware support supervision.      #
########################################################################
Do you want to continue (Y/N)?

0) _Yes
1) _No

Select a number from 0-1: 0
Phase 1: Checking VMFS header and resource files
   Detected VMFS-6 file system (labeled:'nvme2tb') with UUID:68e4cab1-0a865c28-49c0-04ab182311d3, Version 6:82
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0
[root@esxi:~]

esxcli storage filesystem rescanを実行すると、ファイルシステムかスナップショットのどちらかにVMFS領域が認識されている

[root@esxi:~] esxcli storage filesystem rescan
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                 UUID                                 Mounted  Type            Size          Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  ------------  ------------
/vmfs/volumes/68cad69a-e82d8e40-5b65-5bb7fb6107f2  datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2     true  VMFS-6  118380036096   91743059968
/vmfs/volumes/68cad69a-d23fb18e-73e5-5bb7fb6107f2  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2     true  VMFSOS  128580583424  125363552256
/vmfs/volumes/fa8a25f7-ba40ebee-45ac-f419c9f388e0  BOOTBANK1                                   fa8a25f7-ba40ebee-45ac-f419c9f388e0     true  vfat      4293591040    4022075392
/vmfs/volumes/f43b0450-7e4d6762-c6be-52e6552cc1f8  BOOTBANK2                                   f43b0450-7e4d6762-c6be-52e6552cc1f8     true  vfat      4293591040    4021354496
[root@esxi:~] esxcli storage vmfs snapshot lis
Error: Unknown command or namespace storage vmfs snapshot lis

[root@esxi:~] esxcli storage vmfs snapshot list
68e4cab1-0a865c28-49c0-04ab182311d3
   Volume Name: nvme2tb
   VMFS UUID: 68e4cab1-0a865c28-49c0-04ab182311d3
   Can mount: true
   Reason for un-mountability:
   Can resignature: true
   Reason for non-resignaturability:
   Unresolved Extent Count: 1
[root@esxi:~]

今回はスナップショットとして認識されていたので、再署名を行う

[root@esxi:~] esxcli storage vmfs snapshot resignature --volume-label=nvme2tb
[root@esxi:~]

再署名すると”snap”という名前ながら普通のファイルシステムとして認識された

[root@esxi:~] esxcli storage vmfs snapshot list
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                 UUID                                 Mounted  Type             Size           Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  -------------  -------------
/vmfs/volumes/68cad69a-e82d8e40-5b65-5bb7fb6107f2  datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2     true  VMFS-6   118380036096    91743059968
/vmfs/volumes/68e5b682-56352c06-7c60-04ab182311d3  snap-444b0642-nvme2tb                       68e5b682-56352c06-7c60-04ab182311d3     true  VMFS-6  2048162529280  1222844088320
/vmfs/volumes/68cad69a-d23fb18e-73e5-5bb7fb6107f2  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2     true  VMFSOS   128580583424   125363552256
/vmfs/volumes/fa8a25f7-ba40ebee-45ac-f419c9f388e0  BOOTBANK1                                   fa8a25f7-ba40ebee-45ac-f419c9f388e0     true  vfat       4293591040     4022075392
/vmfs/volumes/f43b0450-7e4d6762-c6be-52e6552cc1f8  BOOTBANK2                                   f43b0450-7e4d6762-c6be-52e6552cc1f8     true  vfat       4293591040     4021354496
[root@esxi:~]

再起動しても認識状態は変わらず、普通のVMFS領域として使用できたので、データストア名を元に戻して再使用を開始した


ここから下は調査ログ

ここから下は、状況調査する際に参照した情報について列挙したメモです


KB「VMware ESXi/ESX を操作するときのディスクの識別」にあるコマンドをいくつか実行してみる

[root@esxi:~] esxcli storage core path list
sata.vmhba0-sata.0:1-t10.ATA_____W800S_256GB_____________________________2202211088199_______
   UID: sata.vmhba0-sata.0:1-t10.ATA_____W800S_256GB_____________________________2202211088199_______
   Runtime Name: vmhba0:C0:T1:L0
   Device: t10.ATA_____W800S_256GB_____________________________2202211088199_______
   Device Display Name: Local ATA Disk (t10.ATA_____W800S_256GB_____________________________2202211088199_______)
   Adapter: vmhba0
   Controller: Not Applicable
   Channel: 0
   Target: 1
   LUN: 0
   Plugin: HPP
   State: active
   Transport: sata
   Adapter Identifier: sata.vmhba0
   Target Identifier: sata.0:1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
   Maximum IO Size: 33554432

pcie.300-pcie.0:0-t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   UID: pcie.300-pcie.0:0-t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   Runtime Name: vmhba1:C0:T0:L0
   Device: t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   Device Display Name: Local NVMe Disk (t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000)
   Adapter: vmhba1
   Controller: nqn.2014-08.org.nvmexpress_1e4b_SPD_SP700-2TNGH_________________________0901SP7007D00399
   Channel: 0
   Target: 0
   LUN: 0
   Plugin: HPP
   State: active
   Transport: pcie
   Adapter Identifier: pcie.300
   Target Identifier: pcie.0:0
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
   Maximum IO Size: 524288
[root@esxi:~]
[root@esxi:~] esxcfg-mpath -b
t10.ATA_____W800S_256GB_____________________________2202211088199_______ : Local ATA Disk (t10.ATA_____W800S_256GB_____________________________2202211088199_______)
   vmhba0:C0:T1:L0 LUN:0 state:active Local HBA vmhba0 channel 0 target 1

t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000 : Local NVMe Disk (t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000)
   vmhba1:C0:T0:L0 LUN:0 state:active Local HBA vmhba1 channel 0 target 0

[root@esxi:~]
[root@esxi:~] esxcli storage core device list
t10.ATA_____W800S_256GB_____________________________2202211088199_______
   Display Name: Local ATA Disk (t10.ATA_____W800S_256GB_____________________________2202211088199_______)
   Has Settable Display Name: true
   Size: 244198
   Device Type: Direct-Access
   Multipath Plugin: HPP
   Devfs Path: /vmfs/devices/disks/t10.ATA_____W800S_256GB_____________________________2202211088199_______
   Vendor: ATA
   Model: W800S 256GB
   Revision: 3G5A
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unsupported
   Other UIDs: vml.01000000003232303232313130383831393920202020202020573830305320
   Is Shared Clusterwide: false
   Is SAS: false
   Is USB: false
   Is Boot Device: true
   Device Max Queue Depth: 31
   No of outstanding IOs with competing worlds: 31
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   Display Name: Local NVMe Disk (t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000)
   Has Settable Display Name: true
   Size: 1953514
   Device Type: Direct-Access
   Multipath Plugin: HPP
   Devfs Path: /vmfs/devices/disks/t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
   Vendor: NVMe
   Model: SPD SP700-2TNGH
   Revision: SP02203A
   SCSI Level: 0
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: no
   Attached Filters:
   VAAI Status: unsupported
   Other UIDs: vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2
   Is Shared Clusterwide: false
   Is SAS: false
   Is USB: false
   Is Boot Device: false
   Device Max Queue Depth: 1023
   No of outstanding IOs with competing worlds: 32
   Drive Type: physical
   RAID Level: NA
   Number of Physical Drives: 1
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false
[root@esxi:~]
[root@esxi:~] esxcfg-scsidevs -c
Device UID                                                                Device Type      Console Device                                                                                Size      Multipath PluginDisplay Name
t10.ATA_____W800S_256GB_____________________________2202211088199_______  Direct-Access    /vmfs/devices/disks/t10.ATA_____W800S_256GB_____________________________2202211088199_______  244198MB  HPP     Local ATA Disk (t10.ATA_____W800S_256GB_____________________________2202211088199_______)
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000     Direct-Access    /vmfs/devices/disks/t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000     1953514MB HPP     Local NVMe Disk (t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000)
[root@esxi:~]

vmfsに関する出力となると、2TBデバイスが登場しない

[root@esxi:~] esxcli storage vmfs extent list
Volume Name                                 VMFS UUID                            Extent Number  Device Name                                                               Partition
------------------------------------------  -----------------------------------  -------------  ------------------------------------------------------------------------  ---------
datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2              0  t10.ATA_____W800S_256GB_____________________________2202211088199_______          8
OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2              0  t10.ATA_____W800S_256GB_____________________________2202211088199_______          7
[root@esxi:~]
[root@esxi:~] esxcfg-scsidevs -m
t10.ATA_____W800S_256GB_____________________________2202211088199_______:8 /vmfs/devices/disks/t10.ATA_____W800S_256GB_____________________________2202211088199_______:8 68cad69a-e82d8e40-5b65-5bb7fb6107f2  0  datastore1
t10.ATA_____W800S_256GB_____________________________2202211088199_______:7 /vmfs/devices/disks/t10.ATA_____W800S_256GB_____________________________2202211088199_______:7 68cad69a-d23fb18e-73e5-5bb7fb6107f2  0  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2
[root@esxi:~]
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                 UUID                                 Mounted  Type            Size          Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  ------------  ------------
/vmfs/volumes/68cad69a-e82d8e40-5b65-5bb7fb6107f2  datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2     true  VMFS-6  118380036096   91743059968
/vmfs/volumes/68cad69a-d23fb18e-73e5-5bb7fb6107f2  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2     true  VMFSOS  128580583424  125363552256
/vmfs/volumes/fa8a25f7-ba40ebee-45ac-f419c9f388e0  BOOTBANK1                                   fa8a25f7-ba40ebee-45ac-f419c9f388e0     true  vfat      4293591040    4022075392
/vmfs/volumes/f43b0450-7e4d6762-c6be-52e6552cc1f8  BOOTBANK2                                   f43b0450-7e4d6762-c6be-52e6552cc1f8     true  vfat      4293591040    4021354496
[root@esxi:~]

/vmfs/devices/disks の下を見てみる

[root@esxi:~] ls -alh /vmfs/devices/disks
total 4500908025
drwxr-xr-x    2 root     root         512 Oct  8 00:11 .
drwxr-xr-x   16 root     root         512 Oct  8 00:11 ..
-rw-------    1 root     root      238.5G Oct  8 00:11 t10.ATA_____W800S_256GB_____________________________2202211088199_______
-rw-------    1 root     root      100.0M Oct  8 00:11 t10.ATA_____W800S_256GB_____________________________2202211088199_______:1
-rw-------    1 root     root        4.0G Oct  8 00:11 t10.ATA_____W800S_256GB_____________________________2202211088199_______:5
-rw-------    1 root     root        4.0G Oct  8 00:11 t10.ATA_____W800S_256GB_____________________________2202211088199_______:6
-rw-------    1 root     root      119.9G Oct  8 00:11 t10.ATA_____W800S_256GB_____________________________2202211088199_______:7
-rw-------    1 root     root      110.5G Oct  8 00:11 t10.ATA_____W800S_256GB_____________________________2202211088199_______:8
-rw-------    1 root     root        1.9T Oct  8 00:11 t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
-rw-------    1 root     root        1.9T Oct  8 00:11 t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000:1
lrwxrwxrwx    1 root     root          69 Oct  8 00:11 vml.0100000000303230305f303030305f303030305f3030303000535044205350 -> t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
lrwxrwxrwx    1 root     root          71 Oct  8 00:11 vml.0100000000303230305f303030305f303030305f3030303000535044205350:1 -> t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000:1
lrwxrwxrwx    1 root     root          72 Oct  8 00:11 vml.01000000003232303232313130383831393920202020202020573830305320 -> t10.ATA_____W800S_256GB_____________________________2202211088199_______
lrwxrwxrwx    1 root     root          74 Oct  8 00:11 vml.01000000003232303232313130383831393920202020202020573830305320:1 -> t10.ATA_____W800S_256GB_____________________________2202211088199_______:1
lrwxrwxrwx    1 root     root          74 Oct  8 00:11 vml.01000000003232303232313130383831393920202020202020573830305320:5 -> t10.ATA_____W800S_256GB_____________________________2202211088199_______:5
lrwxrwxrwx    1 root     root          74 Oct  8 00:11 vml.01000000003232303232313130383831393920202020202020573830305320:6 -> t10.ATA_____W800S_256GB_____________________________2202211088199_______:6
lrwxrwxrwx    1 root     root          74 Oct  8 00:11 vml.01000000003232303232313130383831393920202020202020573830305320:7 -> t10.ATA_____W800S_256GB_____________________________2202211088199_______:7
lrwxrwxrwx    1 root     root          74 Oct  8 00:11 vml.01000000003232303232313130383831393920202020202020573830305320:8 -> t10.ATA_____W800S_256GB_____________________________2202211088199_______:8
lrwxrwxrwx    1 root     root          69 Oct  8 00:11 vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2 -> t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
lrwxrwxrwx    1 root     root          71 Oct  8 00:11 vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2:1 -> t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000:1
[root@esxi:~]

KB「Detach a LUN device from ESXi hosts」より

[root@esxi:~] esxcli storage core device world list
Device                                                                    World ID  Open Count  World Name
------------------------------------------------------------------------  --------  ----------  ----------
t10.ATA_____W800S_256GB_____________________________2202211088199_______    524300           1  idle0
t10.ATA_____W800S_256GB_____________________________2202211088199_______    524399           1  OCFlush
t10.ATA_____W800S_256GB_____________________________2202211088199_______    524403           1  bcflushd
t10.ATA_____W800S_256GB_____________________________2202211088199_______    524728           1  Vol3JournalExtendMgrWorld
t10.ATA_____W800S_256GB_____________________________2202211088199_______    524813           1  J6AsyncReplayManager
t10.ATA_____W800S_256GB_____________________________2202211088199_______    525311           1  hostd
t10.ATA_____W800S_256GB_____________________________2202211088199_______    525543           1  healthd
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000       525311           1  hostd
[root@esxi:~] esxcli storage core device world list -d vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2
Device                                                                 World ID  Open Count  World Name
---------------------------------------------------------------------  --------  ----------  ----------
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000    525311           1  hostd
[root@esxi:~]
[root@esxi:~] esxcli storage core adapter list
HBA Name  Driver     Link State  UID          Capabilities  Description
--------  ---------  ----------  -----------  ------------  -----------
vmhba0    vmw_ahci   link-n/a    sata.vmhba0                (0000:00:17.0) Intel Corporation Alder Lake-N SATA AHCI Controller
vmhba1    nvme_pcie  link-n/a    pcie.300                   (0000:03:00.0) MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1602 (DRAM-less)
[root@esxi:~]
[root@esxi:~] esxcli storage core device partition list
Device                                                                    Partition  Start Sector  End Sector  Type           Size
------------------------------------------------------------------------  ---------  ------------  ----------  ----  -------------
t10.ATA_____W800S_256GB_____________________________2202211088199_______          0             0   500118191     0   256060514304
t10.ATA_____W800S_256GB_____________________________2202211088199_______          1            64      204863     0      104857600
t10.ATA_____W800S_256GB_____________________________2202211088199_______          5        208896     8595455     6     4293918720
t10.ATA_____W800S_256GB_____________________________2202211088199_______          6       8597504    16984063     6     4293918720
t10.ATA_____W800S_256GB_____________________________2202211088199_______          7      16986112   268435455    f8   128742064128
t10.ATA_____W800S_256GB_____________________________2202211088199_______          8     268437504   500118158    fb   118620495360
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000             0             0  4000797359     0  2048408248320
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000             1          2048  4000794624    fb  2048405799424
[root@esxi:~] esxcli storage core device partition showguid
Device                                                                    Partition  Layout  GUID
------------------------------------------------------------------------  ---------  ------  ----
t10.ATA_____W800S_256GB_____________________________2202211088199_______          0  GPT     00000000000000000000000000000000
t10.ATA_____W800S_256GB_____________________________2202211088199_______          1  GPT     c12a7328f81f11d2ba4b00a0c93ec93b
t10.ATA_____W800S_256GB_____________________________2202211088199_______          5  GPT     ebd0a0a2b9e5443387c068b6b72699c7
t10.ATA_____W800S_256GB_____________________________2202211088199_______          6  GPT     ebd0a0a2b9e5443387c068b6b72699c7
t10.ATA_____W800S_256GB_____________________________2202211088199_______          7  GPT     4eb2ea3978554790a79efae495e21f8d
t10.ATA_____W800S_256GB_____________________________2202211088199_______          8  GPT     aa31e02a400f11db9590000c2911d1b8
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000             0  GPT     00000000000000000000000000000000
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000             1  GPT     aa31e02a400f11db9590000c2911d1b8
[root@esxi:~]

ESXiのvmkernelのモジュールに関するパラメータを調査

まず、nvmeに関連しそうなモジュール一覧

[root@esxi:~] esxcli system module list|grep nvme
vmknvme                              true        true
vmknvme_vmkapi_compat                true        true
nvme_pcie                            true        true
[root@esxi:~]

それぞれのモジュールにあるパラメータを確認

[root@esxi:~] esxcli system module parameters list --module=nvme_pcie
Name                         Type  Value  Description
---------------------------  ----  -----  -----------
nvmePCIEBlkSizeAwarePollAct  int          NVMe PCIe block size aware poll activate. Valid if poll activated. Default activated.
nvmePCIEDebugMask            int          NVMe PCIe driver debug mask
nvmePCIEDma4KSwitch          int          NVMe PCIe 4k-alignment DMA
nvmePCIEFakeAdminQSize       uint         NVMe PCIe fake ADMIN queue size. 0's based
nvmePCIELogLevel             int          NVMe PCIe driver log level
nvmePCIEMsiEnbaled           int          NVMe PCIe MSI interrupt enable
nvmePCIEPollAct              int          NVMe PCIe hybrid poll activate, MSIX interrupt must be enabled. Default activated.
nvmePCIEPollInterval         uint         NVMe PCIe hybrid poll interval between each poll in microseconds. Valid if poll activated. Default 50us.
nvmePCIEPollOIOThr           uint         NVMe PCIe hybrid poll OIO threshold of automatic switch from interrupt to poll. Valid if poll activated. Default 30 OIO commands per IO queue.
[root@esxi:~] esxcli system module parameters list --module=vmknvme
Name                                   Type  Value  Description
-------------------------------------  ----  -----  -----------
vmknvme_adapter_num_cmpl_queues        uint         Number of PSA completion queues for NVMe-oF adapter, min: 1, max: 16, default: 4
vmknvme_bind_intr                      uint         If enabled, the interrupt cookies are binded to completion worlds. This parameter is only applied when using driver completion worlds.
vmknvme_compl_world_type               uint         completion world type, PSA: 0, VMKNVME: 1
vmknvme_ctlr_recover_initial_attempts  uint         Number of initial controller recover attempts, MIN: 2, MAX: 30
vmknvme_ctlr_recover_method            uint         controller recover method after initial recover attempts, RETRY: 0, DELETE: 1
vmknvme_cw_rate                        uint         Number of completion worlds per IO queue (NVMe/PCIe only). Number is a power of 2. Applies when number of queues less than 4.
vmknvme_enable_noiob                   uint         If enabled, driver will split the commands based on NOIOB.
vmknvme_hostnqn_format                 uint         HostNQN format, UUID: 0, HostName: 1
vmknvme_io_queue_num                   uint         vmknvme IO queue number for NVMe/PCIe adapter: pow of 2 in [1, 16]
vmknvme_io_queue_size                  uint         IO queue size: [8, 1024]
vmknvme_iosplit_workaround             uint         If enabled, qdepth in PSA layer is half size of vmknvme settings.
vmknvme_log_level                      uint         log level: [0, 20]
vmknvme_max_prp_entries_num            uint         User defined maximum number of PRP entries per controller:default value is 0
vmknvme_stats                          uint         Nvme statistics per controller (NVMe/PCIe only now). Logical OR of flags for collecting. 0x0 for disable, 0x1 for basic data (IO pattern), 0x2 for histogram without IO block size, 0x4 for histogram with IO block size. Default 0x2.
vmknvme_total_io_queue_size            uint         Aggregated IO queue size of a controller, MIN: 64, MAX: 4096
vmknvme_use_default_domain_name        uint         If set to 1, the default domain name "com.vmware", not the system domain name will always be used to generate host NQN. Not used: 0, used: 1, default: 0
[root@esxi:~] esxcli system module parameters list --module=vmknvme_vmkapi_compat
[root@esxi:~]

データストアとしての取り扱いに関連しそうなものはなさそうに見える

esxcliを調べるとesxcli nvmeコマンド群があった

HPE Alletra 9000:VMware ESXi実装ガイドのESXiホストからのネームスペースの検出とネームスペースへの接続 に NVMe over FC時のesxcli nvmeコマンドでの実行例があるので実行してみる

[root@esxi:~] esxcli nvme adapter list
Adapter  Adapter Qualified Name                                                               Transport Type  Driver     Associated Devices
-------  -----------------------------------------------------------------------------------  --------------  ---------  ------------------
vmhba1   aqn:nvme_pcie:nqn.2014-08.org.nvmexpress1e4b1e4b0901SP7007D00399____SPD_SP700-2TNGH  PCIe            nvme_pcie
[root@esxi:~]

ネームスペースはすでにある

[root@esxi:~] esxcli nvme namespace list
Name                                                                   Controller Number  Namespace ID  Block Size  Capacity in MB
---------------------------------------------------------------------  -----------------  ------------  ----------  --------------
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000                256             1         512         1953514
[root@esxi:~] esxcli nvme controller list
Name                                                                                      Controller Number  Adapter  Transport Type  Is Online  Controller Type  Is VVOL  Keep Alive Timeout  IO Queue Number  IO Queue Size
----------------------------------------------------------------------------------------  -----------------  -------  --------------  ---------  ---------------  -------  ------------------  ---------------  -------------
nqn.2014-08.org.nvmexpress_1e4b_SPD_SP700-2TNGH_________________________0901SP7007D00399                256  vmhba1   PCIe                 true                     false                   0                1           1024
[root@esxi:~]

PowerEdge:DellサーバーおよびVMware ESXiでのNVMe LED管理 にLED管理の前段階となるデバイスにどういう設定ができるか表示するといった項目があった

[root@esxi:~] esxcli nvme device list
HBA Name  Status  Signature
--------  ------  ---------
vmhba1    Online  nvmeMgmt-nvmhba0
[root@esxi:~] esxcli nvme device get -A vmhba1
Controller Identify Info:
   PCIVID: 0x1e4b
   PCISSVID: 0x1e4b
   Serial Number: 0901SP7007D00399
   Model Number: SPD SP700-2TNGH
   Firmware Revision: SP02203A
   Recommended Arbitration Burst: 0
   IEEE OUI Identifier: 000000
   Controller Associated with an SR-IOV Virtual Function: false
   Controller Associated with a PCI Function: true
   NVM Subsystem May Contain Two or More Controllers: false
   NVM Subsystem Contains Only One Controller: true
   NVM Subsystem May Contain Two or More PCIe Ports: false
   NVM Subsystem Contains Only One PCIe Port: true
   Max Data Transfer Size: 7
   Controller ID: 0
   Version: 1.4
   RTD3 Resume Latency: 500000 us
   RTD3 Entry Latency: 2000000 us
   Optional Firmware Activation Event Support: true
   Optional Namespace Attribute Changed Event Support: false
   Host Identifier Support: false
   Namespace Management and Attachment Support: false
   Firmware Activate and Download Support: true
   Format NVM Support: true
   Security Send and Receive Support: true
   Abort Command Limit: 2
   Async Event Request Limit: 3
   Firmware Activate Without Reset Support: true
   Firmware Slot Number: 3
   The First Slot Is Read-only: false
   Telemetry Log Page Support: false
   Command Effects Log Page Support: true
   SMART/Health Information Log Page per Namespace Support: false
   Error Log Page Entries: 63
   Number of Power States Support: 4
   Format of Admin Vendor Specific Commands Is Same: true
   Format of Admin Vendor Specific Commands Is Vendor Specific: false
   Autonomous Power State Transitions Support: true
   Warning Composite Temperature Threshold: 363
   Critical Composite Temperature Threshold: 368
   Max Time for Firmware Activation: 200 * 100ms
   Host Memory Buffer Preferred Size: 8192 * 4KB
   Host Memory Buffer Min Size: 8192 * 4KB
   Total NVM Capacity: 0x1dceea56000
   Unallocated NVM Capacity: 0x0
   Access Size: 0 * 512B
   Total Size: 0 * 128KB
   Authentication Method: 0
   Number of RPMB Units: 0
   Keep Alive Support: 0
   Max Submission Queue Entry Size: 64 Bytes
   Required Submission Queue Entry Size: 64 Bytes
   Max Completion Queue Entry Size: 16 Bytes
   Required Completion Queue Entry Size: 16 Bytes
   Max Outstanding Commands: 0
   Number of Namespaces: 1
   Reservation Support: false
   Save/Select Field in Set/Get Feature Support: true
   Write Zeroes Command Support: true
   Dataset Management Command Support: true
   Write Uncorrectable Command Support: true
   Compare Command Support: true
   Fused Operation Support: false
   Cryptographic Erase as Part of Secure Erase Support: false
   Cryptographic Erase and User Data Erase to All Namespaces: false
   Cryptographic Erase and User Data Erase to One Particular Namespace: true
   Format Operation to All Namespaces: false
   Format Opertaion to One Particular Namespace: true
   Volatile Write Cache Is Present: true
   Atomic Write Unit Normal: 0 Logical Blocks
   Atomic Write Unit Power Fail: 0 Logical Blocks
   Format of All NVM Vendor Specific Commands Is Same: false
   Format of All NVM Vendor Specific Commands Is Vendor Specific: true
   Atomic Compare and Write Unit: 0
   SGL Address Specify Offset Support: false
   MPTR Contain SGL Descriptor Support: false
   SGL Length Able to Larger than Data Amount: false
   SGL Length Shall Be Equal to Data Amount: true
   Byte Aligned Contiguous Physical Buffer of Metadata Support: false
   SGL Bit Bucket Descriptor Support: false
   SGL Keyed SGL Data Block Descriptor Support: false
   SGL for NVM Command Set Support: false
   NVM Subsystem NVMe Qualified Name:
   NVM Subsystem NVMe Qualified Name (hex format):
[root@esxi:~]

SCSIからNVMe VMware VMFSデータストアへのオフライン移行手順

[root@esxi:~] esxcli storage vmfs lockmode list
Volume Name                                 UUID                                 Type      Locking Mode  ATS Compatible  ATS Upgrade Modes  ATS Incompatibility Reason
------------------------------------------  -----------------------------------  --------  ------------  --------------  -----------------  --------------------------
datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2  VMFS-6    ATS+SCSI               false  None               Device does not support ATS
OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2  Non-VMFS  ATS+SCSI               false  None               Device does not support ATS
[root@esxi:~]

vomaコマンドでファイルシステムチェック

[root@esxi:~] ls /vmfs/devices/disks/
t10.ATA_____W800S_256GB_____________________________2202211088199_______
t10.ATA_____W800S_256GB_____________________________2202211088199_______:1
t10.ATA_____W800S_256GB_____________________________2202211088199_______:5
t10.ATA_____W800S_256GB_____________________________2202211088199_______:6
t10.ATA_____W800S_256GB_____________________________2202211088199_______:7
t10.ATA_____W800S_256GB_____________________________2202211088199_______:8
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000
t10.NVMe____SPD_SP7002D2TNGH_________________________0200000000000000:1
vml.0100000000303230305f303030305f303030305f3030303000535044205350
vml.0100000000303230305f303030305f303030305f3030303000535044205350:1
vml.01000000003232303232313130383831393920202020202020573830305320
vml.01000000003232303232313130383831393920202020202020573830305320:1
vml.01000000003232303232313130383831393920202020202020573830305320:5
vml.01000000003232303232313130383831393920202020202020573830305320:6
vml.01000000003232303232313130383831393920202020202020573830305320:7
vml.01000000003232303232313130383831393920202020202020573830305320:8
vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2
vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b89f9e2:1
[root@esxi:~] voma -m vmfs -f check -N -d /vmfs/devices/disks/vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b8
9f9e2:1
Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done

Checking for filesystem activity
Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
         Reservation Support is not present for NVME devices
Performing filesystem liveness check..|
########################################################################
#   Warning !!!                                                        #
#                                                                      #
#   You are about to execute VOMA without device reservation.          #
#   Any access to this device from other hosts when VOMA is running    #
#   can cause severe data corruption                                   #
#                                                                      #
#   This mode is supported only under VMware support supervision.      #
########################################################################
Do you want to continue (Y/N)?

0) _Yes
1) _No

Select a number from 0-1: 0
Phase 1: Checking VMFS header and resource files
   Detected VMFS-6 file system (labeled:'nvme2tb') with UUID:68e4cab1-0a865c28-49c0-04ab182311d3, Version 6:82
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0
[root@esxi:~]

ファイルシステムが追加されたわけではない?

[root@esxi:~] esxcli storage filesystem rescan
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                 UUID                                 Mounted  Type            Size          Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  ------------  ------------
/vmfs/volumes/68cad69a-e82d8e40-5b65-5bb7fb6107f2  datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2     true  VMFS-6  118380036096   91743059968
/vmfs/volumes/68cad69a-d23fb18e-73e5-5bb7fb6107f2  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2     true  VMFSOS  128580583424  125363552256
/vmfs/volumes/fa8a25f7-ba40ebee-45ac-f419c9f388e0  BOOTBANK1                                   fa8a25f7-ba40ebee-45ac-f419c9f388e0     true  vfat      4293591040    4022075392
/vmfs/volumes/f43b0450-7e4d6762-c6be-52e6552cc1f8  BOOTBANK2                                   f43b0450-7e4d6762-c6be-52e6552cc1f8     true  vfat      4293591040    4021354496
[root@esxi:~]

スナップショットがある?

[root@esxi:~] esxcli storage vmfs snapshot list
68e4cab1-0a865c28-49c0-04ab182311d3
   Volume Name: nvme2tb
   VMFS UUID: 68e4cab1-0a865c28-49c0-04ab182311d3
   Can mount: true
   Reason for un-mountability:
   Can resignature: true
   Reason for non-resignaturability:
   Unresolved Extent Count: 1
[root@esxi:~]

再署名してみた

[root@esxi:~] esxcli storage vmfs snapshot resignature --volume-label=nvme2tb
[root@esxi:~] esxcli storage vmfs snapshot list
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                 UUID                                 Mounted  Type             Size           Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  -------------  -------------
/vmfs/volumes/68cad69a-e82d8e40-5b65-5bb7fb6107f2  datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2     true  VMFS-6   118380036096    91743059968
/vmfs/volumes/68e5b682-56352c06-7c60-04ab182311d3  snap-444b0642-nvme2tb                       68e5b682-56352c06-7c60-04ab182311d3     true  VMFS-6  2048162529280  1222844088320
/vmfs/volumes/68cad69a-d23fb18e-73e5-5bb7fb6107f2  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2     true  VMFSOS   128580583424   125363552256
/vmfs/volumes/fa8a25f7-ba40ebee-45ac-f419c9f388e0  BOOTBANK1                                   fa8a25f7-ba40ebee-45ac-f419c9f388e0     true  vfat       4293591040     4022075392
/vmfs/volumes/f43b0450-7e4d6762-c6be-52e6552cc1f8  BOOTBANK2                                   f43b0450-7e4d6762-c6be-52e6552cc1f8     true  vfat       4293591040     4021354496
[root@esxi:~]

スナップショット領域がファイルシステムとして認識された?

名前がsnapとついてるだけで普通のデータストア?

もう1回vomaでチェック

[root@esxi:~] voma -m vmfs -f check -N -d /vmfs/devices/disks/vml.05c56298c6cae09f64ef49957d1d7af93c98b2a5792c87d191b47f87ea5b8
9f9e2:1
Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done

Checking for filesystem activity
Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
         Reservation Support is not present for NVME devices
Performing filesystem liveness check..|
########################################################################
#   Warning !!!                                                        #
#                                                                      #
#   You are about to execute VOMA without device reservation.          #
#   Any access to this device from other hosts when VOMA is running    #
#   can cause severe data corruption                                   #
#                                                                      #
#   This mode is supported only under VMware support supervision.      #
########################################################################
Do you want to continue (Y/N)?

0) _Yes
1) _No

Select a number from 0-1: 0
Phase 1: Checking VMFS header and resource files
   Detected VMFS-6 file system (labeled:'snap-444b0642-nvme2tb') with UUID:68e5b682-56352c06-7c60-04ab182311d3, Version 6:82
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0
[root@esxi:~] esxcli storage vmfs snapshot list
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                 UUID                                 Mounted  Type             Size           Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  -------------  -------------
/vmfs/volumes/68cad69a-e82d8e40-5b65-5bb7fb6107f2  datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2     true  VMFS-6   118380036096    91743059968
/vmfs/volumes/68e5b682-56352c06-7c60-04ab182311d3  snap-444b0642-nvme2tb                       68e5b682-56352c06-7c60-04ab182311d3     true  VMFS-6  2048162529280  1222844088320
/vmfs/volumes/68cad69a-d23fb18e-73e5-5bb7fb6107f2  OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2     true  VMFSOS   128580583424   125363552256
/vmfs/volumes/fa8a25f7-ba40ebee-45ac-f419c9f388e0  BOOTBANK1                                   fa8a25f7-ba40ebee-45ac-f419c9f388e0     true  vfat       4293591040     4022075392
/vmfs/volumes/f43b0450-7e4d6762-c6be-52e6552cc1f8  BOOTBANK2                                   f43b0450-7e4d6762-c6be-52e6552cc1f8     true  vfat       4293591040     4021354496
[root@esxi:~]

特に状況は変わらない

lockmode確認すると、そちらでもデバイスは増えた

[root@esxi:~] esxcli storage vmfs lockmode list
Volume Name                                 UUID                                 Type      Locking Mode  ATS Compatible  ATS Upgrade Modes  ATS Incompatibility Reason
------------------------------------------  -----------------------------------  --------  ------------  --------------  -----------------  --------------------------
datastore1                                  68cad69a-e82d8e40-5b65-5bb7fb6107f2  VMFS-6    ATS+SCSI               false  None               Device does not support ATS
snap-444b0642-nvme2tb                       68e5b682-56352c06-7c60-04ab182311d3  VMFS-6    ATS+SCSI               false  None               Device does not support ATS
OSDATA-68cad69a-d23fb18e-73e5-5bb7fb6107f2  68cad69a-d23fb18e-73e5-5bb7fb6107f2  Non-VMFS  ATS+SCSI               false  None               Device does not support ATS
[root@esxi:~]

とりあえずESXiを再起動

再起動してみても、同じ認識状況だったので、snapを普通の名前に変えて使用継続することとした

All FlashのNetAppでノード1側にディスクを片寄せする場合のメモ

All FlashのnetApp AFF C250は通常2つあるノードに対して均等にディスクが割り当てられている。

ONTAP公式ドキュメント:ルート/データパーティショニング

例えば15TB SSDを10本つんでいる場合、以下のようになる

各スロットにある15TB SSDは内部で、data1パーテーション(約7TB)、data2パーテーション(約7TB)、rootパーテーション(残り) に分割されるという、ADPv2というフォーマットになっている。

オレンジ色の部分のownerがノード1で、水色の部分のownerがノード2となる。

data1パーテーションとdata2パーテーションのownerをノード1に割り当てなおすと以下のような構成をとることが可能となり、ボリュームを拡大することができる。

owner変更してノード1側に集約しよう、という場合は、ノード2側に作成されているaggregateは壊す必要があるので注意

作業で使うコマンドについて

パーテーション変更を行う際に、owner状態などを確認する必要がある。

その際に使うコマンドには、通常権限(admin)で実行できるものと、diag権限で実行できるものがある。

各パーテーションのownerがどちらのノードなのか確認するコマンド

通常権限コマンド「storage disk show -partition-ownership」

通常権限コマンド「storage disk show -fields owner,type,root-owner,data1-owner,data2-owner」

diag権限コマンド「storage disk partition show」

パーテーションの割り当てを変更するときに、変更忘れがないか確認する場合は「storage disk partition show -partition *P2」パーテーションを指定してOwnerノードの表示を一括で確認するとよい

パーテーションのownerを変更するコマンド

パーテーションownerを変更するコマンドはdiag権限の「storage disk partition assign」で行う

「storage disk partition assign -partition *P2 -owner ノード1 -force」と実行すると有無も言わさず強制的にすべてのP2パーテーション(data2パーテーション)のownerをノード1に変更することができるが、事故を起こさないように事前に「storage disk partition show -partition *P2」を実行し、すべての”Container Type”が「spare」であることを確認した方がよい

すでにaggregateが作成されている場合

すでに両ノードでaggregateが作成されている場合、ownerをはく奪するノード側のaggregateを削除する必要がある。

なお、システム用のaggregate(標準ではaggr0_ノード名 で作成されている) は削除しないこと。

削除手順は「storage aggregate offline -aggregate aggr名」でオフラインにした後

「storage aggregate delete -aggregate aggr名」で削除する。


ADPv2のパーテーション配分について

15TB SSDが8本の場合、各ノードのrootパーテーション部分は4パーテーションを使用したRAID-DPとなりスペアはない状態となる。

各ノードが使うシステム用aggregateは140GB程度は確保されるのだが、ディスクの本数が少ない場合、rootパーテーションに割り当てられる容量が通常より増やされれていた。

data1data2rootrootのスペア
8本6.94TB6.94TB93.52GBなし
10本6.94TB6.94TB93.52GB1
12本6.96TB6.96TB62.35GB1
14本6.96TB6.96TB62.35GB1
16本~22本6.97TB6.97TB37.42GB1
24本6.97TB6.97TB37.42GB2