テスト用に作ったceph環境でOSDが落ちまくるので osd heartbeat grace を変更してみた(様子見中

Proxmox VEのcephストレージ環境の動作を確認するためESXi上に RAM 16GBの仮想マシンを4台作ってテスト中(+1台 corosync qnetdサーバがいてProxmox VEクラスタの維持に使用)

で、あるタイミングから、各ノード上のosdのdownが多発するようになった

1root@pve37:~# ceph osd df tree
2ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
3-1         0.62549         -  480 GiB  210 GiB  204 GiB  178 KiB  5.4 GiB  270 GiB  43.65  1.00    -          root default
4-3         0.15637         -  160 GiB   59 GiB   58 GiB   47 KiB  1.4 GiB  101 GiB  36.92  0.85    -              host pve36
5 0    hdd  0.03909   1.00000   40 GiB   15 GiB   14 GiB    7 KiB  403 MiB   25 GiB  36.27  0.83   36      up          osd.0
6 1    hdd  0.03909   1.00000   40 GiB   18 GiB   17 GiB   13 KiB  332 MiB   22 GiB  44.03  1.01   52      up          osd.1
7 2    hdd  0.03909   1.00000   40 GiB   11 GiB   10 GiB   18 KiB  337 MiB   29 GiB  26.46  0.61   27      up          osd.2
8 3    hdd  0.03909   1.00000   40 GiB   16 GiB   16 GiB    9 KiB  393 MiB   24 GiB  40.91  0.94   48      up          osd.3
9-5         0.15637         -  160 GiB   67 GiB   66 GiB   75 KiB  1.6 GiB   93 GiB  41.95  0.96    -              host pve37
10 4    hdd  0.03909   1.00000   40 GiB   19 GiB   18 GiB   24 KiB  443 MiB   21 GiB  46.87  1.07   51      up          osd.4
11 5    hdd  0.03909   1.00000   40 GiB   11 GiB   11 GiB   21 KiB  201 MiB   29 GiB  28.58  0.65   30      up          osd.5
12 6    hdd  0.03909   1.00000   40 GiB   16 GiB   16 GiB   12 KiB  294 MiB   24 GiB  39.51  0.91   40      up          osd.6
13 7    hdd  0.03909   1.00000   40 GiB   21 GiB   20 GiB   18 KiB  693 MiB   19 GiB  52.84  1.21   61      up          osd.7
14-7         0.15637         -   80 GiB   49 GiB   47 GiB   36 KiB  1.3 GiB   31 GiB  60.91  1.40    -              host pve38
15 8    hdd  0.03909         0      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.8
16 9    hdd  0.03909         0      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.9
1710    hdd  0.03909   1.00000   40 GiB   20 GiB   20 GiB   17 KiB  415 MiB   20 GiB  51.02  1.17   53      up          osd.10
1811    hdd  0.03909   1.00000   40 GiB   28 GiB   27 GiB   19 KiB  922 MiB   12 GiB  70.80  1.62   73      up          osd.11
19-9         0.15637         -   80 GiB   35 GiB   34 GiB   20 KiB  1.1 GiB   45 GiB  43.27  0.99    -              host pve39
2012    hdd  0.03909   1.00000   40 GiB   20 GiB   20 GiB    7 KiB  824 MiB   20 GiB  50.81  1.16   63      up          osd.12
2113    hdd  0.03909         0      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.13
2214    hdd  0.03909   1.00000   40 GiB   14 GiB   14 GiB   13 KiB  303 MiB   26 GiB  35.72  0.82    0    down          osd.14
2315    hdd  0.03909         0      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.15
24                       TOTAL  480 GiB  210 GiB  204 GiB  183 KiB  5.4 GiB  270 GiB  43.65
25MIN/MAX VAR: 0.61/1.62  STDDEV: 11.55
26root@pve37:~#

pve38のosd.8とosd.9がdownになっているので、pve38にログインしてプロセスを確認すると、–id 8 と –id 9のceph-osdサービスが起動していないので、これらを再起動する

1root@pve38:~# ps -ef|grep osd
2ceph        1676       1  1 12:14 ?        00:02:01 /usr/bin/ceph-osd -f --cluster ceph --id 10 --setuser ceph --setgroup ceph
3ceph        1681       1  2 12:14 ?        00:02:45 /usr/bin/ceph-osd -f --cluster ceph --id 11 --setuser ceph --setgroup ceph
4root       30916   30893  0 14:10 pts/0    00:00:00 grep osd
5root@pve38:~# systemctl restart ceph-osd@8
6root@pve38:~# systemctl restart ceph-osd@9
7root@pve38:~#

しばらく待つとupになる

1root@pve38:~# ceph osd df tree
2ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
3-1         0.62549         -  600 GiB  227 GiB  221 GiB  229 KiB  5.6 GiB  373 GiB  37.84  1.00    -          root default
4-3         0.15637         -  160 GiB   53 GiB   52 GiB   47 KiB  1.4 GiB  107 GiB  33.11  0.88    -              host pve36
5 0    hdd  0.03909   1.00000   40 GiB   13 GiB   12 GiB    7 KiB  403 MiB   27 GiB  31.50  0.83   28      up          osd.0
6 1    hdd  0.03909   1.00000   40 GiB   16 GiB   16 GiB   13 KiB  332 MiB   24 GiB  40.30  1.07   47      up          osd.1
7 2    hdd  0.03909   1.00000   40 GiB  9.7 GiB  9.4 GiB   18 KiB  337 MiB   30 GiB  24.21  0.64   22      up          osd.2
8 3    hdd  0.03909   1.00000   40 GiB   15 GiB   14 GiB    9 KiB  393 MiB   25 GiB  36.41  0.96   41      up          osd.3
9-5         0.15637         -  160 GiB   61 GiB   59 GiB   75 KiB  1.6 GiB   99 GiB  37.89  1.00    -              host pve37
10 4    hdd  0.03909   1.00000   40 GiB   16 GiB   15 GiB   24 KiB  443 MiB   24 GiB  39.75  1.05   41      up          osd.4
11 5    hdd  0.03909   1.00000   40 GiB   10 GiB   10 GiB   21 KiB  201 MiB   30 GiB  25.52  0.67   26      up          osd.5
12 6    hdd  0.03909   1.00000   40 GiB   14 GiB   13 GiB   12 KiB  278 MiB   26 GiB  34.26  0.91   32      up          osd.6
13 7    hdd  0.03909   1.00000   40 GiB   21 GiB   20 GiB   18 KiB  693 MiB   19 GiB  52.02  1.37   52      up          osd.7
14-7         0.15637         -  120 GiB   57 GiB   55 GiB   54 KiB  1.5 GiB   63 GiB  47.17  1.25    -              host pve38
15 8    hdd  0.03909   1.00000   40 GiB   14 GiB   14 GiB   18 KiB  132 MiB   26 GiB  35.75  0.94   30      up          osd.8
16 9    hdd  0.03909   1.00000      0 B      0 B      0 B      0 B      0 B      0 B      0     0   22      up          osd.9
1710    hdd  0.03909   1.00000   40 GiB   18 GiB   18 GiB   17 KiB  419 MiB   22 GiB  45.92  1.21   42      up          osd.10
1811    hdd  0.03909   1.00000   40 GiB   24 GiB   23 GiB   19 KiB  939 MiB   16 GiB  59.84  1.58   42      up          osd.11
19-9         0.15637         -  160 GiB   57 GiB   56 GiB   53 KiB  1.2 GiB  103 GiB  35.51  0.94    -              host pve39
2012    hdd  0.03909   1.00000   40 GiB   15 GiB   14 GiB    7 KiB  841 MiB   25 GiB  37.05  0.98   37      up          osd.12
2113    hdd  0.03909   1.00000   40 GiB   16 GiB   16 GiB   16 KiB  144 MiB   24 GiB  39.70  1.05   42      up          osd.13
2214    hdd  0.03909   1.00000   40 GiB   14 GiB   14 GiB   16 KiB   84 MiB   26 GiB  35.82  0.95   39      up          osd.14
2315    hdd  0.03909   1.00000   40 GiB   12 GiB   12 GiB   14 KiB  127 MiB   28 GiB  29.48  0.78   30      up          osd.15
24                       TOTAL  600 GiB  227 GiB  221 GiB  236 KiB  5.6 GiB  373 GiB  37.84
25MIN/MAX VAR: 0/1.58  STDDEV: 12.91
26root@pve38:~#

が・・・またしばらくすると、他のosdが落ちる、などしていた

RedHat Ceph Storage 7 トラブルシューティングガイドの「第5章 Ceph OSD のトラブルシューティング」5.1.7. OSDS のフラップを確認すると、osdに指定されているディスクが遅いから、ということになるようだ。

osd_heartbeat_grace_time というパラメータをデフォルトの20秒から変更すると、タイムアウトまでの値を緩和できるのかな、と思ったのだが、どうやって設定するのかが不明…

ceph.orgのOSD Setting を見ると /etc/ceph/ceph.conf (PVEの場合、 /etc/pve/ceph.conf )に追加すればいいのかな？というところなんだけど、OSD Config Reference , Configuring Monitor/OSD Interaction を見ても osd_heartbeat_grace_time というパラメータが無い…(osd_heartbeat_grace ならあった)

RedHatドキュメントの続きに書いてある「この問題を解決するには、以下を行います。」のところを見ると、「ceph osd set noup」「ceph osd set nodown」を設定して、OSDをdownおよびupとしてマークするのを停止する、とある。

試しにnoup,nodownの療法を設定してみたところ、OSDサービスを起動してもceph osd df treeで確認するとdownのままとなっていた。

まあ、upになったとしてもupのマークを付けないのが「noup」だから当然ですね・・・

そんなわけで、「ceph osd unset noup」「ceph osd set nodown」でdownにしない、という設定を入れてみた

設定を入れると「ceph osd stat」での状態確認で「flags nodown」と表示されるようになる。

1root@pve38:~# ceph osd stat
216 osds: 16 up (since 62m), 16 in (since 62m); epoch: e4996
3flags nodown
4root@pve38:~#

とりあえず、これで一時的なごまかしはできた。

ただ、これは、OSDで使用しているディスクが壊れたとしても downにならない、ということでもある。

なので、「nodown」フラグを設定しっぱなしで使う、というのはとても不適切となる。

ちゃんとした対処を行うためには、具体的に何が問題になっているのかを「ceph health detail」を実行して、具体的にSlow OSD heartbeats がどれくらい遅いのかを確認する

1root@pve38:~# ceph health detail
2HEALTH_WARN nodown flag(s) set; Slow OSD heartbeats on back (longest 5166.450ms); Slow OSD heartbeats on front (longest 5467.151ms)
3[WRN] OSDMAP_FLAGS: nodown flag(s) set
4[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 5166.450ms)
5    Slow OSD heartbeats on back from osd.13 [] to osd.8 [] 5166.450 msec
6    Slow OSD heartbeats on back from osd.13 [] to osd.0 [] 3898.044 msec
7    Slow OSD heartbeats on back from osd.12 [] to osd.9 [] 3268.881 msec
8    Slow OSD heartbeats on back from osd.10 [] to osd.3 [] 2610.064 msec possibly improving
9    Slow OSD heartbeats on back from osd.12 [] to osd.8 [] 2588.321 msec
10    Slow OSD heartbeats on back from osd.6 [] to osd.14 [] 2565.141 msec
11    Slow OSD heartbeats on back from osd.8 [] to osd.7 [] 2385.851 msec possibly improving
12    Slow OSD heartbeats on back from osd.13 [] to osd.11 [] 2324.505 msec
13    Slow OSD heartbeats on back from osd.8 [] to osd.12 [] 2305.474 msec possibly improving
14    Slow OSD heartbeats on back from osd.14 [] to osd.11 [] 2275.033 msec
15    Truncated long network list.  Use ceph daemon mgr.# dump_osd_network for more information
16[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 5467.151ms)
17    Slow OSD heartbeats on front from osd.13 [] to osd.8 [] 5467.151 msec
18    Slow OSD heartbeats on front from osd.13 [] to osd.0 [] 3956.364 msec
19    Slow OSD heartbeats on front from osd.12 [] to osd.9 [] 3513.493 msec
20    Slow OSD heartbeats on front from osd.12 [] to osd.8 [] 2657.999 msec
21    Slow OSD heartbeats on front from osd.6 [] to osd.14 [] 2657.486 msec
22    Slow OSD heartbeats on front from osd.10 [] to osd.3 [] 2610.558 msec possibly improving
23    Slow OSD heartbeats on front from osd.8 [] to osd.7 [] 2436.661 msec possibly improving
24    Slow OSD heartbeats on front from osd.14 [] to osd.11 [] 2351.914 msec
25    Slow OSD heartbeats on front from osd.14 [] to osd.10 [] 2351.812 msec
26    Slow OSD heartbeats on front from osd.13 [] to osd.11 [] 2335.698 msec
27    Truncated long network list.  Use ceph daemon mgr.# dump_osd_network for more information
28root@pve38:~#

osd.7のログが出てるpve37にログインして /var/log/ceph/ceph-osd.7.log から「no replay from」と「osd.8」でgrep をかけてログを確認

おそらく「Slow OSD heartbeats on front from osd.8 [] to osd.7 [] 2436.661 msec」に相当するあたりがコレなのかな？というところ

12024-11-14T14:46:05.457+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:02.037605+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)
22024-11-14T14:46:06.454+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:02.037605+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)
32024-11-14T14:46:07.467+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:07.338127+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)
42024-11-14T14:46:08.418+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:07.338127+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)
52024-11-14T14:46:09.371+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:09.038264+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)
62024-11-14T14:46:10.416+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:09.038264+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)
72024-11-14T14:46:11.408+0900 7e72364006c0 -1 osd.7 4996 heartbeat_check: no reply from 172.17.44.38:6802 osd.8 since back 2024-11-14T14:46:11.338592+0900 front 2024-11-14T14:45:41.850539+0900 (oldest deadline 2024-11-14T14:46:05.334473+0900)

oldset deadlineにある時刻と、その前にある時刻の差は20秒なので、 osd_heartbeat_grace もしくは osd_heartbeat_grace_time のデフォルト値 20 が効いてるんだろうなぁ、と推定できる

設定手法について記載を探してみたのだがなかなかない

Ceph Block Device 3rd Party Integration » Ceph iSCSI Gateway » iSCSI Gateway Requirements に下記のような設定例がある

1[osd]
2osd heartbeat grace = 20
3osd heartbeat interval = 5

また、下記のように個別OSDに対して値を設定することも可能であるようだ

`1`	`ceph tell osd.* config set osd_heartbeat_grace 20`

`2`	`ceph tell osd.* config set osd_heartbeat_interval 5`

`1`	`ceph daemon osd.0 config set osd_heartbeat_grace 20`

`2`	`ceph daemon osd.0 config set osd_heartbeat_interval 5`

ceph tellの書式を確認すると「ceph tell osd.* config get osd_heartbeat_grace」で値がとれる模様

1root@pve37:~# ceph tell osd.* config get osd_heartbeat_grace
2osd.0: {
3    "osd_heartbeat_grace": "20"
4}
5osd.1: {
6    "osd_heartbeat_grace": "20"
7}
8osd.2: {
9    "osd_heartbeat_grace": "20"
10}
11osd.3: {
12    "osd_heartbeat_grace": "20"
13}
14osd.4: {
15    "osd_heartbeat_grace": "20"
16}
17osd.5: {
18    "osd_heartbeat_grace": "20"
19}
20osd.6: {
21    "osd_heartbeat_grace": "20"
22}
23osd.7: {
24    "osd_heartbeat_grace": "20"
25}
26osd.8: {
27    "osd_heartbeat_grace": "20"
28}
29osd.9: {
30    "osd_heartbeat_grace": "20"
31}
32osd.10: {
33    "osd_heartbeat_grace": "20"
34}
35osd.11: {
36    "osd_heartbeat_grace": "20"
37}
38osd.12: {
39    "osd_heartbeat_grace": "20"
40}
41osd.13: {
42    "osd_heartbeat_grace": "20"
43}
44osd.14: {
45    "osd_heartbeat_grace": "20"
46}
47osd.15: {
48    "osd_heartbeat_grace": "20"
49}
50root@pve37:~#

とりあえず「ceph tell osd.* config set osd_heartbeat_grace 30」と実行し、30に設定してみる

1root@pve37:~# ceph tell osd.* config set osd_heartbeat_grace 30
2osd.0: {
3    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
4}
5osd.1: {
6    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
7}
8osd.2: {
9    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
10}
11osd.3: {
12    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
13}
14osd.4: {
15    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
16}
17osd.5: {
18    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
19}
20osd.6: {
21    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
22}
23osd.7: {
24    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
25}
26osd.8: {
27    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
28}
29osd.9: {
30    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
31}
32osd.10: {
33    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
34}
35osd.11: {
36    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
37}
38osd.12: {
39    "success": "osd_heartbeat_grace = '' (not observed, change may require restart) "
40}
41osd.13: {
42    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
43}
44osd.14: {
45    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
46}
47osd.15: {
48    "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_heartbeat_grace = '' (not observed, change may require restart) osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' "
49}
50root@pve37:~#

すべて「”success”」ではあるので設定変更は完了しているのだと思うが、応答が2種類あるのはなんなのだろうか？

設定が変更されたかどうかを確認

1root@pve37:~# ceph tell osd.* config get osd_heartbeat_grace
2osd.0: {
3    "osd_heartbeat_grace": "30"
4}
5osd.1: {
6    "osd_heartbeat_grace": "30"
7}
8osd.2: {
9    "osd_heartbeat_grace": "30"
10}
11osd.3: {
12    "osd_heartbeat_grace": "30"
13}
14osd.4: {
15    "osd_heartbeat_grace": "30"
16}
17osd.5: {
18    "osd_heartbeat_grace": "30"
19}
20osd.6: {
21    "osd_heartbeat_grace": "30"
22}
23osd.7: {
24    "osd_heartbeat_grace": "30"
25}
26osd.8: {
27    "osd_heartbeat_grace": "30"
28}
29osd.9: {
30    "osd_heartbeat_grace": "30"
31}
32osd.10: {
33    "osd_heartbeat_grace": "30"
34}
35osd.11: {
36    "osd_heartbeat_grace": "30"
37}
38osd.12: {
39    "osd_heartbeat_grace": "30"
40}
41osd.13: {
42    "osd_heartbeat_grace": "30"
43}
44osd.14: {
45    "osd_heartbeat_grace": "30"
46}
47osd.15: {
48    "osd_heartbeat_grace": "30"
49}
50root@pve37:~#

とはいえ、set時の出力に「(not observed, change may require restart)」とあるとおり、ceph-osdの再起動が必須であるようだ

/etc/pve/ceph.conf に変更したパラメータは反映されてない模様なので、 osd.4～osd.7があるサーバを再起動してからもう一度値を確認してみたら、20に戻っていた。

1root@pve38:~# ceph tell osd.* config get osd_heartbeat_grace
2osd.0: {
3    "osd_heartbeat_grace": "30"
4}
5osd.1: {
6    "osd_heartbeat_grace": "30"
7}
8osd.2: {
9    "osd_heartbeat_grace": "30"
10}
11osd.3: {
12    "osd_heartbeat_grace": "30"
13}
14osd.4: {
15    "osd_heartbeat_grace": "20"
16}
17osd.5: {
18    "osd_heartbeat_grace": "20"
19}
20osd.6: {
21    "osd_heartbeat_grace": "20"
22}
23osd.7: {
24    "osd_heartbeat_grace": "20"
25}
26osd.8: {
27    "osd_heartbeat_grace": "30"
28}
29osd.9: {
30    "osd_heartbeat_grace": "30"
31}
32osd.10: {
33    "osd_heartbeat_grace": "30"
34}
35osd.11: {
36    "osd_heartbeat_grace": "30"
37}
38osd.12: {
39    "osd_heartbeat_grace": "30"
40}
41osd.13: {
42    "osd_heartbeat_grace": "30"
43}
44osd.14: {
45    "osd_heartbeat_grace": "30"
46}
47osd.15: {
48    "osd_heartbeat_grace": "30"
49}
50root@pve38:~#

/etc/pve/ceph.conf の最後に下記を追加

1 [osd]

`2`	`osd heartbeat grace = 30`

設定後、再起動してから確認すると、想定通り30になっているのを確認。そもそも、osd_heartbeat_grace についてはceph tellコマンドでの設定変更後、再起動しないでも大丈夫、というやつなんでは？

1root@pve38:~# ceph tell osd.* config get osd_heartbeat_grace
2osd.0: {
3    "osd_heartbeat_grace": "30"
4}
5osd.1: {
6    "osd_heartbeat_grace": "30"
7}
8osd.2: {
9    "osd_heartbeat_grace": "30"
10}
11osd.3: {
12    "osd_heartbeat_grace": "30"
13}
14osd.4: {
15    "osd_heartbeat_grace": "30"
16}
17osd.5: {
18    "osd_heartbeat_grace": "30"
19}
20osd.6: {
21    "osd_heartbeat_grace": "30"
22}
23osd.7: {
24    "osd_heartbeat_grace": "30"
25}
26osd.8: {
27    "osd_heartbeat_grace": "30"
28}
29osd.9: {
30    "osd_heartbeat_grace": "30"
31}
32osd.10: {
33    "osd_heartbeat_grace": "30"
34}
35osd.11: {
36    "osd_heartbeat_grace": "30"
37}
38osd.12: {
39    "osd_heartbeat_grace": "30"
40}
41osd.13: {
42    "osd_heartbeat_grace": "30"
43}
44osd.14: {
45    "osd_heartbeat_grace": "30"
46}
47osd.15: {
48    "osd_heartbeat_grace": "30"
49}
50root@pve38:~#

コメントを残す コメントをキャンセル

コメントを残すコメントをキャンセル