Starting / Stopping Ceph at Redmond
Jump to navigation
Jump to search
root@str-237:~# systemctl start ceph.target
root@str-238:~# systemctl start ceph.target
WAIT at LEAST 60S
root@str-238:~# ceph -s
cluster:
id: 75b74760-4e61-11ed-b975-0b6dbf7a8a34
health: HEALTH_WARN
1 hosts fail cephadm check
pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set
services:
mon: 2 daemons, quorum str-238,str-237 (age 5m)
mgr: str-238.aujped(active, since 5m), standbys: str-237.aicbzp
osd: 24 osds: 24 up (since 4m), 24 in (since 11M)
flags pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover
data:
pools: 6 pools, 385 pgs
objects: 22.34M objects, 24 TiB
usage: 62 TiB used, 106 TiB / 168 TiB avail
pgs: 385 active+clean
root@str-238:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 167.67114 root default
-9 83.83557 host str-237
1 nvme 6.98630 osd.1 up 1.00000 1.00000
5 nvme 6.98630 osd.5 up 1.00000 1.00000
9 nvme 6.98630 osd.9 up 1.00000 1.00000
13 nvme 6.98630 osd.13 up 1.00000 1.00000
17 nvme 6.98630 osd.17 up 1.00000 1.00000
21 nvme 6.98630 osd.21 up 1.00000 1.00000
25 nvme 6.98630 osd.25 up 1.00000 1.00000
29 nvme 6.98630 osd.29 up 1.00000 1.00000
33 nvme 6.98630 osd.33 up 1.00000 1.00000
37 nvme 6.98630 osd.37 up 1.00000 1.00000
41 nvme 6.98630 osd.41 up 1.00000 1.00000
45 nvme 6.98630 osd.45 up 1.00000 1.00000
-7 83.83557 host str-238
2 nvme 6.98630 osd.2 up 1.00000 1.00000
7 nvme 6.98630 osd.7 up 1.00000 1.00000
11 nvme 6.98630 osd.11 up 1.00000 1.00000
15 nvme 6.98630 osd.15 up 1.00000 1.00000
19 nvme 6.98630 osd.19 up 1.00000 1.00000
23 nvme 6.98630 osd.23 up 1.00000 1.00000
27 nvme 6.98630 osd.27 up 1.00000 1.00000
31 nvme 6.98630 osd.31 up 1.00000 1.00000
35 nvme 6.98630 osd.35 up 1.00000 1.00000
39 nvme 6.98630 osd.39 up 1.00000 1.00000
43 nvme 6.98630 osd.43 up 1.00000 1.00000
47 nvme 6.98630 osd.47 up 1.00000 1.00000
-3 0 host str-239
-5 0 host str-249
LOTS OF THIS IS A LIE because of all the flags DO NOT BE a Shuji CHECK BEFORE YOU UNSET THEM
There are 12 osds on each node and 1 mon
We’re going to check every one by hand
root@str-238:~# systemctl status ceph-*
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@mon.str-238.service - Ceph mon.str-238 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165654 (bash)
Tasks: 26 (limit: 618873)
Memory: 17.8M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@mon.str-238.service
├─165654 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/mon.str-238/unit.run
└─166109 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-mon-str-238 -e CONTAINER_IMA>
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.626+0000 7fe036f58700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625680629391, "job": 18, "event": "table_file_deletion", "file_number": 881559}
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe036f58700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-str-238/store.db/881557.sst immediately, rate_bytes_per_sec 0, total_tra>
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe036f58700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625680632540, "job": 18, "event": "table_file_deletion", "file_number": 881557}
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe02d745700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe02d745700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe02d745700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe02d745700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting
Sep 13 17:21:20 str-238 bash[166109]: debug 2023-09-13T17:21:20.630+0000 7fe02d745700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting
Sep 13 17:21:21 str-238 bash[166109]: cluster 2023-09-13T17:21:20.011286+0000 mgr.str-238.aujped (mgr.15644118) 193 : cluster [DBG] pgmap v193: 385 pgs: 385 active+clean; 24 TiB data, 62 TiB used, 106 TiB / 168 TiB avail
Sep 13 17:21:23 str-238 bash[166109]: cluster 2023-09-13T17:21:22.012221+0000 mgr.str-238.aujped (mgr.15644118) 194 : cluster [DBG] pgmap v194: 385 pgs: 385 active+clean; 24 TiB data, 62 TiB used, 106 TiB / 168 TiB avail
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.39.service - Ceph osd.39 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165747 (bash)
Tasks: 27 (limit: 618873)
Memory: 21.6M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.39.service
├─165747 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.39/unit.run
└─248161 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-39 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023766, "job": 18, "event": "table_file_deletion", "file_number": 10499}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023823, "job": 18, "event": "table_file_deletion", "file_number": 10498}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023898, "job": 18, "event": "table_file_deletion", "file_number": 10497}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023975, "job": 18, "event": "table_file_deletion", "file_number": 10496}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359024057, "job": 18, "event": "table_file_deletion", "file_number": 10495}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359024125, "job": 18, "event": "table_file_deletion", "file_number": 10494}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359024199, "job": 18, "event": "table_file_deletion", "file_number": 10493}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: (Original Log Time 2023/09/13-17:15:59.024220) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do
Sep 13 17:16:12 str-238 bash[248161]: debug 2023-09-13T17:16:12.792+0000 7f5474456700 0 log_channel(cluster) log [DBG] : 3.11 scrub starts
Sep 13 17:16:12 str-238 bash[248161]: debug 2023-09-13T17:16:12.796+0000 7f5474456700 0 log_channel(cluster) log [DBG] : 3.11 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.2.service - Ceph osd.2 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165715 (bash)
Tasks: 24 (limit: 618873)
Memory: 20.0M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.2.service
├─165715 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.2/unit.run
└─249291 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-2 -e CONTAINER_IMAGE=qua>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7375700 1 osd.2 pg_epoch: 30208 pg[2.25( v 30078'786055 (29895'778087,30078'786055] local-lis/les=28504/28505 n=87429 ec=83/83 lis/c=28504/28504 les/c/f=2850>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7b76700 1 osd.2 pg_epoch: 30208 pg[2.c( v 30042'2309520 (29906'2301530,30042'2309520] local-lis/les=28657/28658 n=86943 ec=83/83 lis/c=28657/28657 les/c/f=28>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7b76700 1 osd.2 pg_epoch: 30208 pg[2.c( v 30042'2309520 (29906'2301530,30042'2309520] local-lis/les=28657/28658 n=86943 ec=83/83 lis/c=28657/28657 les/c/f=28>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.8e( v 30110'948583 (29926'941759,30110'948583] local-lis/les=27772/27773 n=86733 ec=83/83 lis/c=27772/27772 les/c/f=2777>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.8e( v 30110'948583 (29926'941759,30110'948583] local-lis/les=27772/27773 n=86733 ec=83/83 lis/c=27772/27772 les/c/f=2777>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7b76700 1 osd.2 pg_epoch: 30208 pg[2.c( v 30042'2309520 (29906'2301530,30042'2309520] local-lis/les=28657/28658 n=86943 ec=83/83 lis/c=28657/28657 les/c/f=28>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.440+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.46( v 30008'818872 (29926'812745,30008'818872] local-lis/les=28297/28298 n=86608 ec=83/83 lis/c=28297/28297 les/c/f=2829>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.440+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.46( v 30008'818872 (29926'812745,30008'818872] local-lis/les=28297/28298 n=86608 ec=83/83 lis/c=28297/28297 les/c/f=2829>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.684+0000 7fc6e236b700 0 log_channel(cluster) log [DBG] : 2.5f scrub starts
Sep 13 17:16:11 str-238 bash[249291]: debug 2023-09-13T17:16:11.984+0000 7fc6e236b700 0 log_channel(cluster) log [DBG] : 2.5f scrub ok
root@str-238:~# systemctl status ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd*
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.39.service - Ceph osd.39 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165747 (bash)
Tasks: 27 (limit: 618873)
Memory: 21.6M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.39.service
├─165747 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.39/unit.run
└─248161 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-39 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023766, "job": 18, "event": "table_file_deletion", "file_number": 10499}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023823, "job": 18, "event": "table_file_deletion", "file_number": 10498}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023898, "job": 18, "event": "table_file_deletion", "file_number": 10497}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359023975, "job": 18, "event": "table_file_deletion", "file_number": 10496}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359024057, "job": 18, "event": "table_file_deletion", "file_number": 10495}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359024125, "job": 18, "event": "table_file_deletion", "file_number": 10494}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625359024199, "job": 18, "event": "table_file_deletion", "file_number": 10493}
Sep 13 17:15:59 str-238 bash[248161]: debug 2023-09-13T17:15:59.020+0000 7f5484476700 4 rocksdb: (Original Log Time 2023/09/13-17:15:59.024220) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do
Sep 13 17:16:12 str-238 bash[248161]: debug 2023-09-13T17:16:12.792+0000 7f5474456700 0 log_channel(cluster) log [DBG] : 3.11 scrub starts
Sep 13 17:16:12 str-238 bash[248161]: debug 2023-09-13T17:16:12.796+0000 7f5474456700 0 log_channel(cluster) log [DBG] : 3.11 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.2.service - Ceph osd.2 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165715 (bash)
Tasks: 24 (limit: 618873)
Memory: 20.0M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.2.service
├─165715 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.2/unit.run
└─249291 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-2 -e CONTAINER_IMAGE=qua>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7375700 1 osd.2 pg_epoch: 30208 pg[2.25( v 30078'786055 (29895'778087,30078'786055] local-lis/les=28504/28505 n=87429 ec=83/83 lis/c=28504/28504 les/c/f=2850>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7b76700 1 osd.2 pg_epoch: 30208 pg[2.c( v 30042'2309520 (29906'2301530,30042'2309520] local-lis/les=28657/28658 n=86943 ec=83/83 lis/c=28657/28657 les/c/f=28>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7b76700 1 osd.2 pg_epoch: 30208 pg[2.c( v 30042'2309520 (29906'2301530,30042'2309520] local-lis/les=28657/28658 n=86943 ec=83/83 lis/c=28657/28657 les/c/f=28>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.8e( v 30110'948583 (29926'941759,30110'948583] local-lis/les=27772/27773 n=86733 ec=83/83 lis/c=27772/27772 les/c/f=2777>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.8e( v 30110'948583 (29926'941759,30110'948583] local-lis/les=27772/27773 n=86733 ec=83/83 lis/c=27772/27772 les/c/f=2777>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.436+0000 7fc6e7b76700 1 osd.2 pg_epoch: 30208 pg[2.c( v 30042'2309520 (29906'2301530,30042'2309520] local-lis/les=28657/28658 n=86943 ec=83/83 lis/c=28657/28657 les/c/f=28>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.440+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.46( v 30008'818872 (29926'812745,30008'818872] local-lis/les=28297/28298 n=86608 ec=83/83 lis/c=28297/28297 les/c/f=2829>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.440+0000 7fc6e6b74700 1 osd.2 pg_epoch: 30208 pg[2.46( v 30008'818872 (29926'812745,30008'818872] local-lis/les=28297/28298 n=86608 ec=83/83 lis/c=28297/28297 les/c/f=2829>
Sep 13 17:15:57 str-238 bash[249291]: debug 2023-09-13T17:15:57.684+0000 7fc6e236b700 0 log_channel(cluster) log [DBG] : 2.5f scrub starts
Sep 13 17:16:11 str-238 bash[249291]: debug 2023-09-13T17:16:11.984+0000 7fc6e236b700 0 log_channel(cluster) log [DBG] : 2.5f scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.15.service - Ceph osd.15 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165692 (bash)
Tasks: 22 (limit: 618873)
Memory: 20.7M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.15.service
├─165692 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.15/unit.run
└─247754 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-15 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:56 str-238 bash[247754]: debug 2023-09-13T17:15:56.508+0000 7f4a9a79d700 0 log_channel(cluster) log [DBG] : 3.19 scrub starts
Sep 13 17:15:56 str-238 bash[247754]: debug 2023-09-13T17:15:56.512+0000 7f4a9a79d700 0 log_channel(cluster) log [DBG] : 3.19 scrub ok
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9cfa2700 1 osd.15 pg_epoch: 30208 pg[2.34( v 30040'986464 (29895'979612,30040'986464] local-lis/les=28398/28399 n=87901 ec=83/83 lis/c=28398/28398 les/c/f=283>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9d7a3700 1 osd.15 pg_epoch: 30208 pg[2.cb( v 30062'1006101 (29926'999210,30062'1006101] local-lis/les=27369/27370 n=87326 ec=83/83 lis/c=27369/27369 les/c/f=2>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9cfa2700 1 osd.15 pg_epoch: 30208 pg[2.34( v 30040'986464 (29895'979612,30040'986464] local-lis/les=28398/28399 n=87901 ec=83/83 lis/c=28398/28398 les/c/f=283>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9d7a3700 1 osd.15 pg_epoch: 30208 pg[2.cb( v 30062'1006101 (29926'999210,30062'1006101] local-lis/les=27369/27370 n=87326 ec=83/83 lis/c=27369/27369 les/c/f=2>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9cfa2700 1 osd.15 pg_epoch: 30208 pg[2.34( v 30040'986464 (29895'979612,30040'986464] local-lis/les=28398/28399 n=87901 ec=83/83 lis/c=28398/28398 les/c/f=283>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9d7a3700 1 osd.15 pg_epoch: 30208 pg[2.cb( v 30062'1006101 (29926'999210,30062'1006101] local-lis/les=27369/27370 n=87326 ec=83/83 lis/c=27369/27369 les/c/f=2>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.436+0000 7f4a9af9e700 1 osd.15 pg_epoch: 30208 pg[2.d8( v 30021'1262851 (29926'1256424,30021'1262851] local-lis/les=27282/27283 n=86870 ec=83/83 lis/c=27282/27282 les/c/f=>
Sep 13 17:15:57 str-238 bash[247754]: debug 2023-09-13T17:15:57.440+0000 7f4a9af9e700 1 osd.15 pg_epoch: 30208 pg[2.d8( v 30021'1262851 (29926'1256424,30021'1262851] local-lis/les=27282/27283 n=86870 ec=83/83 lis/c=27282/27282 les/c/f=>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.35.service - Ceph osd.35 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165740 (bash)
Tasks: 21 (limit: 618873)
Memory: 20.7M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.35.service
├─165740 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.35/unit.run
└─251959 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-35 -e CONTAINER_IMAGE=qu>
Sep 13 17:16:31 str-238 bash[251959]: debug 2023-09-13T17:16:31.224+0000 7fe8ee277700 0 log_channel(cluster) log [DBG] : 6.c scrub starts
Sep 13 17:16:31 str-238 bash[251959]: debug 2023-09-13T17:16:31.308+0000 7fe8f227f700 0 log_channel(cluster) log [DBG] : 6.c scrub ok
Sep 13 17:16:33 str-238 bash[251959]: debug 2023-09-13T17:16:33.216+0000 7fe8eca74700 0 log_channel(cluster) log [DBG] : 4.1f scrub starts
Sep 13 17:16:33 str-238 bash[251959]: debug 2023-09-13T17:16:33.220+0000 7fe8f0a7c700 0 log_channel(cluster) log [DBG] : 4.1f scrub ok
Sep 13 17:16:34 str-238 bash[251959]: debug 2023-09-13T17:16:34.212+0000 7fe8eda76700 0 log_channel(cluster) log [DBG] : 3.5 scrub starts
Sep 13 17:16:40 str-238 bash[251959]: debug 2023-09-13T17:16:40.176+0000 7fe8f227f700 0 log_channel(cluster) log [DBG] : 5.c scrub starts
Sep 13 17:16:40 str-238 bash[251959]: debug 2023-09-13T17:16:40.180+0000 7fe8f227f700 0 log_channel(cluster) log [DBG] : 5.c scrub ok
Sep 13 17:16:43 str-238 bash[251959]: debug 2023-09-13T17:16:43.152+0000 7fe8eda76700 0 log_channel(cluster) log [DBG] : 3.5 scrub starts
Sep 13 17:16:44 str-238 bash[251959]: debug 2023-09-13T17:16:44.129+0000 7fe8eda76700 0 log_channel(cluster) log [DBG] : 3.5 scrub starts
Sep 13 17:16:44 str-238 bash[251959]: debug 2023-09-13T17:16:44.137+0000 7fe8f1a7e700 0 log_channel(cluster) log [DBG] : 3.5 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.27.service - Ceph osd.27 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165722 (bash)
Tasks: 25 (limit: 618873)
Memory: 21.0M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.27.service
├─165722 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.27/unit.run
└─251177 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-27 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.436+0000 7f99e431c700 1 osd.27 pg_epoch: 30208 pg[2.83( v 30097'772009 (29920'765820,30097'772009] local-lis/les=27863/27864 n=87597 ec=83/83 lis/c=27863/27863 les/c/f=278>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.436+0000 7f99e2b19700 1 osd.27 pg_epoch: 30208 pg[2.ee( v 30066'1151992 (29926'1146382,30066'1151992] local-lis/les=27135/27136 n=87710 ec=83/83 lis/c=27135/27135 les/c/f=>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.436+0000 7f99e531e700 1 osd.27 pg_epoch: 30208 pg[2.a1( v 30058'1201218 (29926'1195570,30058'1201218] local-lis/les=27657/27658 n=87534 ec=83/83 lis/c=27657/27657 les/c/f=>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.436+0000 7f99e6320700 1 osd.27 pg_epoch: 30208 pg[5.1f( v 30128'6898 (11251'6609,30128'6898] local-lis/les=16835/16836 n=25 ec=97/97 lis/c=16835/16835 les/c/f=16836/16836/>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.440+0000 7f99e2b19700 1 osd.27 pg_epoch: 30208 pg[2.ee( v 30066'1151992 (29926'1146382,30066'1151992] local-lis/les=27135/27136 n=87710 ec=83/83 lis/c=27135/27135 les/c/f=>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.440+0000 7f99e6320700 1 osd.27 pg_epoch: 30208 pg[5.1f( v 30128'6898 (11251'6609,30128'6898] local-lis/les=16835/16836 n=25 ec=97/97 lis/c=16835/16835 les/c/f=16836/16836/>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.440+0000 7f99e6320700 1 osd.27 pg_epoch: 30208 pg[3.17( empty local-lis/les=16785/16786 n=0 ec=94/94 lis/c=16785/16785 les/c/f=16786/16786/0 sis=30208) [39,27,31] r=1 lpr=>
Sep 13 17:15:57 str-238 bash[251177]: debug 2023-09-13T17:15:57.440+0000 7f99e6320700 1 osd.27 pg_epoch: 30208 pg[3.17( empty local-lis/les=16785/16786 n=0 ec=94/94 lis/c=16785/16785 les/c/f=16786/16786/0 sis=30208) [39,27,31] r=1 lpr=>
Sep 13 17:16:13 str-238 bash[251177]: debug 2023-09-13T17:16:13.052+0000 7f99e7b23700 0 log_channel(cluster) log [DBG] : 2.bc scrub starts
Sep 13 17:16:29 str-238 bash[251177]: debug 2023-09-13T17:16:29.100+0000 7f99e7b23700 0 log_channel(cluster) log [DBG] : 2.bc scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.19.service - Ceph osd.19 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165710 (bash)
Tasks: 21 (limit: 618873)
Memory: 20.7M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.19.service
├─165710 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.19/unit.run
└─249409 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-19 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7faba04e2700 1 osd.19 pg_epoch: 30207 pg[2.ed( v 30058'3504723 (29926'3498274,30058'3504723] local-lis/les=27139/27140 n=87214 ec=83/83 lis/c=27139/27139 les/c/f=>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7fab9ccdb700 1 osd.19 pg_epoch: 30207 pg[2.ac( v 30111'1697948 (29926'1691483,30111'1697948] local-lis/les=27585/27586 n=86855 ec=83/83 lis/c=27585/27585 les/c/f=>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7fab9fce1700 1 osd.19 pg_epoch: 30207 pg[2.6( v 30030'1134230 (29926'1126461,30030'1134230] local-lis/les=28700/28701 n=86828 ec=83/83 lis/c=28700/28700 les/c/f=2>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7faba14e4700 1 osd.19 pg_epoch: 30207 pg[2.63( v 30097'976385 (29926'969990,30097'976385] local-lis/les=28066/28067 n=86989 ec=83/83 lis/c=28066/28066 les/c/f=280>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7faba14e4700 1 osd.19 pg_epoch: 30207 pg[2.63( v 30097'976385 (29926'969990,30097'976385] local-lis/les=28066/28067 n=86989 ec=83/83 lis/c=28066/28066 les/c/f=280>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7fab9d4dc700 1 osd.19 pg_epoch: 30207 pg[2.eb( v 30122'827678 (29926'821281,30122'827678] local-lis/les=27147/27148 n=87667 ec=83/83 lis/c=27147/27147 les/c/f=271>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7fab9d4dc700 1 osd.19 pg_epoch: 30207 pg[2.eb( v 30122'827678 (29926'821281,30122'827678] local-lis/les=27147/27148 n=87667 ec=83/83 lis/c=27147/27147 les/c/f=271>
Sep 13 17:15:56 str-238 bash[249409]: debug 2023-09-13T17:15:56.432+0000 7fab9fce1700 1 osd.19 pg_epoch: 30207 pg[2.6( v 30030'1134230 (29926'1126461,30030'1134230] local-lis/les=28700/28701 n=86828 ec=83/83 lis/c=28700/28700 les/c/f=2>
Sep 13 17:15:57 str-238 bash[249409]: debug 2023-09-13T17:15:57.436+0000 7fab9dcdd700 1 osd.19 pg_epoch: 30208 pg[2.da( v 30122'5491131 (29926'5484970,30122'5491131] local-lis/les=27273/27274 n=87053 ec=83/83 lis/c=27273/27273 les/c/f=>
Sep 13 17:15:57 str-238 bash[249409]: debug 2023-09-13T17:15:57.436+0000 7fab9dcdd700 1 osd.19 pg_epoch: 30208 pg[2.da( v 30122'5491131 (29926'5484970,30122'5491131] local-lis/les=27273/27274 n=87053 ec=83/83 lis/c=27273/27273 les/c/f=>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.7.service - Ceph osd.7 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165779 (bash)
Tasks: 25 (limit: 618873)
Memory: 20.6M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.7.service
├─165779 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.7/unit.run
└─251316 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-7 -e CONTAINER_IMAGE=qua>
Sep 13 17:16:00 str-238 bash[251316]: debug 2023-09-13T17:16:00.476+0000 7f5dc08c8700 4 rocksdb: (Original Log Time 2023/09/13-17:16:00.482877) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do
Sep 13 17:16:08 str-238 bash[251316]: debug 2023-09-13T17:16:08.672+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:10 str-238 bash[251316]: debug 2023-09-13T17:16:10.644+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:28 str-238 bash[251316]: debug 2023-09-13T17:16:28.740+0000 7f5dae8a4700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:46 str-238 bash[251316]: debug 2023-09-13T17:16:46.929+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:53 str-238 bash[251316]: debug 2023-09-13T17:16:53.937+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:57 str-238 bash[251316]: debug 2023-09-13T17:16:57.037+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:59 str-238 bash[251316]: debug 2023-09-13T17:16:59.025+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:16:59 str-238 bash[251316]: debug 2023-09-13T17:16:59.985+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub starts
Sep 13 17:17:00 str-238 bash[251316]: debug 2023-09-13T17:17:00.073+0000 7f5daa89c700 0 log_channel(cluster) log [DBG] : 6.5 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.43.service - Ceph osd.43 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165757 (bash)
Tasks: 24 (limit: 618873)
Memory: 21.1M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.43.service
├─165757 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.43/unit.run
└─248318 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-43 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:57 str-238 bash[248318]: debug 2023-09-13T17:15:57.436+0000 7f209028f700 1 osd.43 pg_epoch: 30208 pg[2.3f( v 30034'1299321 (29926'1293613,30034'1299321] local-lis/les=28326/28327 n=87974 ec=83/83 lis/c=28326/28326 les/c/f=>
Sep 13 17:15:57 str-238 bash[248318]: debug 2023-09-13T17:15:57.436+0000 7f209028f700 1 osd.43 pg_epoch: 30208 pg[2.3f( v 30034'1299321 (29926'1293613,30034'1299321] local-lis/les=28326/28327 n=87974 ec=83/83 lis/c=28326/28326 les/c/f=>
Sep 13 17:15:57 str-238 bash[248318]: debug 2023-09-13T17:15:57.436+0000 7f208f28d700 1 osd.43 pg_epoch: 30208 pg[2.a1( v 30058'1201218 (29926'1195570,30058'1201218] local-lis/les=27657/27658 n=87534 ec=83/83 lis/c=27657/27657 les/c/f=>
Sep 13 17:15:57 str-238 bash[248318]: debug 2023-09-13T17:15:57.436+0000 7f208f28d700 1 osd.43 pg_epoch: 30208 pg[2.a1( v 30058'1201218 (29926'1195570,30058'1201218] local-lis/les=27657/27658 n=87534 ec=83/83 lis/c=27657/27657 les/c/f=>
Sep 13 17:15:58 str-238 bash[248318]: debug 2023-09-13T17:15:58.072+0000 7f208c287700 0 log_channel(cluster) log [DBG] : 2.4f scrub starts
Sep 13 17:16:00 str-238 bash[248318]: debug 2023-09-13T17:16:00.048+0000 7f209028f700 0 log_channel(cluster) log [DBG] : 2.4f scrub starts
Sep 13 17:16:01 str-238 bash[248318]: debug 2023-09-13T17:16:01.980+0000 7f208c287700 0 log_channel(cluster) log [DBG] : 2.4f deep-scrub starts
Sep 13 17:16:11 str-238 bash[248318]: debug 2023-09-13T17:16:11.912+0000 7f208c287700 0 log_channel(cluster) log [DBG] : 2.4f scrub starts
Sep 13 17:16:12 str-238 bash[248318]: debug 2023-09-13T17:16:12.912+0000 7f208c287700 0 log_channel(cluster) log [DBG] : 2.4f scrub starts
Sep 13 17:16:28 str-238 bash[248318]: debug 2023-09-13T17:16:28.332+0000 7f209028f700 0 log_channel(cluster) log [DBG] : 2.4f scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.23.service - Ceph osd.23 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165716 (bash)
Tasks: 24 (limit: 618873)
Memory: 21.0M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.23.service
├─165716 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.23/unit.run
└─248329 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-23 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:57 str-238 bash[248329]: debug 2023-09-13T17:15:57.440+0000 7fc5c2fb4700 1 osd.23 pg_epoch: 30208 pg[2.6c( v 30151'657517 (29926'652156,30151'657517] local-lis/les=28015/28016 n=87254 ec=83/83 lis/c=28015/28015 les/c/f=280>
Sep 13 17:15:57 str-238 bash[248329]: debug 2023-09-13T17:15:57.440+0000 7fc5c2fb4700 1 osd.23 pg_epoch: 30208 pg[2.6c( v 30151'657517 (29926'652156,30151'657517] local-lis/les=28015/28016 n=87254 ec=83/83 lis/c=28015/28015 les/c/f=280>
Sep 13 17:15:57 str-238 bash[248329]: debug 2023-09-13T17:15:57.976+0000 7fc5c37b5700 0 log_channel(cluster) log [DBG] : 2.f3 scrub starts
Sep 13 17:16:12 str-238 bash[248329]: debug 2023-09-13T17:16:12.812+0000 7fc5c37b5700 0 log_channel(cluster) log [DBG] : 2.f3 scrub ok
Sep 13 17:16:32 str-238 bash[248329]: debug 2023-09-13T17:16:31.996+0000 7fc5c7fbe700 0 log_channel(cluster) log [DBG] : 2.32 deep-scrub starts
Sep 13 17:18:59 str-238 bash[248329]: debug 2023-09-13T17:18:59.731+0000 7fc5c3fb6700 0 log_channel(cluster) log [DBG] : 2.32 deep-scrub ok
Sep 13 17:19:02 str-238 bash[248329]: debug 2023-09-13T17:19:02.819+0000 7fc5c8fc0700 0 log_channel(cluster) log [DBG] : 2.48 scrub starts
Sep 13 17:19:17 str-238 bash[248329]: debug 2023-09-13T17:19:17.856+0000 7fc5c8fc0700 0 log_channel(cluster) log [DBG] : 2.48 scrub ok
Sep 13 17:19:35 str-238 bash[248329]: debug 2023-09-13T17:19:35.712+0000 7fc5c1fb2700 0 log_channel(cluster) log [DBG] : 6.16 scrub starts
Sep 13 17:19:35 str-238 bash[248329]: debug 2023-09-13T17:19:35.808+0000 7fc5c5fba700 0 log_channel(cluster) log [DBG] : 6.16 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.31.service - Ceph osd.31 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165725 (bash)
Tasks: 21 (limit: 618873)
Memory: 20.3M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.31.service
├─165725 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.31/unit.run
└─250998 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-31 -e CONTAINER_IMAGE=qu>
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364531845, "job": 15, "event": "table_file_deletion", "file_number": 12130}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364531983, "job": 15, "event": "table_file_deletion", "file_number": 12129}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364532133, "job": 15, "event": "table_file_deletion", "file_number": 12128}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364532283, "job": 15, "event": "table_file_deletion", "file_number": 12127}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364532393, "job": 15, "event": "table_file_deletion", "file_number": 12126}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364532475, "job": 15, "event": "table_file_deletion", "file_number": 12125}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1694625364532584, "job": 15, "event": "table_file_deletion", "file_number": 12124}
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: (Original Log Time 2023/09/13-17:16:04.532614) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do
Sep 13 17:16:04 str-238 bash[250998]: debug 2023-09-13T17:16:04.528+0000 7f0ceffff700 4 rocksdb: (Original Log Time 2023/09/13-17:16:04.532626) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do
Sep 13 17:16:12 str-238 bash[250998]: debug 2023-09-13T17:16:12.984+0000 7f0cdffdf700 0 log_channel(cluster) log [DBG] : 2.71 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.11.service - Ceph osd.11 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165667 (bash)
Tasks: 19 (limit: 618873)
Memory: 19.5M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.11.service
├─165667 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.11/unit.run
└─247827 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-11 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:58 str-238 bash[247827]: debug 2023-09-13T17:15:58.504+0000 7f0061693700 0 log_channel(cluster) log [DBG] : 6.d scrub starts
Sep 13 17:16:00 str-238 bash[247827]: debug 2023-09-13T17:16:00.524+0000 7f0062e96700 0 log_channel(cluster) log [DBG] : 2.22 deep-scrub starts
Sep 13 17:16:03 str-238 bash[247827]: debug 2023-09-13T17:16:03.512+0000 7f005d68b700 0 log_channel(cluster) log [DBG] : 6.d scrub starts
Sep 13 17:16:04 str-238 bash[247827]: debug 2023-09-13T17:16:04.512+0000 7f0062e96700 0 log_channel(cluster) log [DBG] : 2.22 scrub starts
Sep 13 17:16:06 str-238 bash[247827]: debug 2023-09-13T17:16:06.472+0000 7f0061693700 0 log_channel(cluster) log [DBG] : 6.d scrub starts
Sep 13 17:16:09 str-238 bash[247827]: debug 2023-09-13T17:16:09.456+0000 7f0062e96700 0 log_channel(cluster) log [DBG] : 2.22 scrub starts
Sep 13 17:16:11 str-238 bash[247827]: debug 2023-09-13T17:16:11.492+0000 7f005d68b700 0 log_channel(cluster) log [DBG] : 6.d scrub starts
Sep 13 17:16:11 str-238 bash[247827]: debug 2023-09-13T17:16:11.588+0000 7f005d68b700 0 log_channel(cluster) log [DBG] : 6.d scrub ok
Sep 13 17:16:16 str-238 bash[247827]: debug 2023-09-13T17:16:16.480+0000 7f005ee8e700 0 log_channel(cluster) log [DBG] : 2.22 scrub starts
Sep 13 17:16:31 str-238 bash[247827]: debug 2023-09-13T17:16:31.232+0000 7f0062e96700 0 log_channel(cluster) log [DBG] : 2.22 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.47.service - Ceph osd.47 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:14:59 UTC; 6min ago
Main PID: 165770 (bash)
Tasks: 23 (limit: 618873)
Memory: 19.4M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.47.service
├─165770 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.47/unit.run
└─248497 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-47 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:56 str-238 bash[248497]: debug 2023-09-13T17:15:56.432+0000 7f28450e6700 1 osd.47 pg_epoch: 30207 pg[6.6( v 8983'4514640 (8970'4513567,8983'4514640] local-lis/les=11565/11566 n=587 ec=3863/3863 lis/c=11565/11565 les/c/f=11>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28418df700 1 osd.47 pg_epoch: 30208 pg[3.d( empty local-lis/les=16877/16878 n=0 ec=94/94 lis/c=16877/16877 les/c/f=16878/16878/0 sis=30207) [47,37] r=0 lpr=3020>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28418df700 1 osd.47 pg_epoch: 30208 pg[3.d( empty local-lis/les=16877/16878 n=0 ec=94/94 lis/c=16877/16877 les/c/f=16878/16878/0 sis=30208) [47,37,31] r=0 lpr=3>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28418df700 1 osd.47 pg_epoch: 30208 pg[3.d( empty local-lis/les=16877/16878 n=0 ec=94/94 lis/c=16877/16877 les/c/f=16878/16878/0 sis=30208) [47,37,31] r=0 lpr=3>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28448e5700 1 osd.47 pg_epoch: 30208 pg[2.ef( v 30062'938952 (29895'932039,30062'938952] local-lis/les=27124/27125 n=86916 ec=83/83 lis/c=27124/27124 les/c/f=271>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28420e0700 1 osd.47 pg_epoch: 30208 pg[2.5c( v 30110'1019317 (29926'1013950,30110'1019317] local-lis/les=28129/28130 n=86615 ec=83/83 lis/c=28129/28129 les/c/f=>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28420e0700 1 osd.47 pg_epoch: 30208 pg[2.5c( v 30110'1019317 (29926'1013950,30110'1019317] local-lis/les=28129/28130 n=86615 ec=83/83 lis/c=28129/28129 les/c/f=>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28448e5700 1 osd.47 pg_epoch: 30208 pg[2.ef( v 30062'938952 (29895'932039,30062'938952] local-lis/les=27124/27125 n=86916 ec=83/83 lis/c=27124/27124 les/c/f=271>
Sep 13 17:15:57 str-238 bash[248497]: debug 2023-09-13T17:15:57.436+0000 7f28448e5700 1 osd.47 pg_epoch: 30208 pg[2.ef( v 30062'938952 (29895'932039,30062'938952] local-lis/les=27124/27125 n=86916 ec=83/83 lis/c=27124/27124 les/c/f=271>
Sep 13 17:21:36 str-238 bash[248497]: debug 2023-09-13T17:21:36.947+0000 7f28438e3700 0 log_channel(cluster) log [DBG] : 2.39 deep-scrub starts
That’s 1 mon and 12 OSD. . . as required all running and none showing weird looking messages in the logs and we would be seeing issues after 7 mons yes you need to wait this long (>5 min) to be sure
Ok other node lets take a look as ze OSDs
root@str-237:~# systemctl status ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd*
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.21.service - Ceph osd.21 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535584 (bash)
Tasks: 23 (limit: 618873)
Memory: 21.5M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.21.service
├─3535584 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.21/unit.run
└─3609266 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-21 -e CONTAINER_IMAGE=q>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a027e700 1 osd.21 pg_epoch: 30208 pg[2.c9( v 30013'632450 (29926'624734,30013'632450] local-lis/les=27380/27381 n=87259 ec=83/83 lis/c=27380/27380 les/c/f=27>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a027e700 1 osd.21 pg_epoch: 30208 pg[2.c9( v 30013'632450 (29926'624734,30013'632450] local-lis/les=27380/27381 n=87259 ec=83/83 lis/c=27380/27380 les/c/f=27>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a027e700 1 osd.21 pg_epoch: 30208 pg[4.11( empty local-lis/les=16184/16185 n=0 ec=95/95 lis/c=16184/16184 les/c/f=16185/16185/0 sis=30207) [21,29] r=0 lpr=30>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a2282700 1 osd.21 pg_epoch: 30208 pg[2.5d( v 30061'1066415 (29926'1060494,30061'1066415] local-lis/les=28125/28126 n=86918 ec=83/83 lis/c=28125/28125 les/c/f>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a027e700 1 osd.21 pg_epoch: 30208 pg[4.11( empty local-lis/les=16184/16185 n=0 ec=95/95 lis/c=16184/16184 les/c/f=16185/16185/0 sis=30208) [21,31,29] r=0 lpr>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a027e700 1 osd.21 pg_epoch: 30208 pg[4.11( empty local-lis/les=16184/16185 n=0 ec=95/95 lis/c=16184/16184 les/c/f=16185/16185/0 sis=30208) [21,31,29] r=0 lpr>
Sep 13 17:15:57 str-237 bash[3609266]: debug 2023-09-13T17:15:57.438+0000 7fc9a2282700 1 osd.21 pg_epoch: 30208 pg[2.5d( v 30061'1066415 (29926'1060494,30061'1066415] local-lis/les=28125/28126 n=86918 ec=83/83 lis/c=28125/28125 les/c/f>
Sep 13 17:16:11 str-237 bash[3609266]: debug 2023-09-13T17:16:11.270+0000 7fc99ea7b700 0 log_channel(cluster) log [DBG] : 2.4 scrub ok
Sep 13 17:22:03 str-237 bash[3609266]: debug 2023-09-13T17:22:03.346+0000 7fc9a4286700 0 log_channel(cluster) log [DBG] : 4.11 scrub starts
Sep 13 17:22:03 str-237 bash[3609266]: debug 2023-09-13T17:22:03.350+0000 7fc9a4286700 0 log_channel(cluster) log [DBG] : 4.11 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.37.service - Ceph osd.37 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535615 (bash)
Tasks: 28 (limit: 618873)
Memory: 22.1M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.37.service
├─3535615 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.37/unit.run
└─3607700 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-37 -e CONTAINER_IMAGE=q>
Sep 13 17:15:57 str-237 bash[3607700]: debug 2023-09-13T17:15:57.434+0000 7f3a365f0700 1 osd.37 pg_epoch: 30208 pg[2.af( v 30012'881315 (29926'875121,30012'881315] local-lis/les=27566/27567 n=87244 ec=83/83 lis/c=27566/27566 les/c/f=27>
Sep 13 17:16:13 str-237 bash[3607700]: debug 2023-09-13T17:16:13.954+0000 7f3a3ddff700 0 log_channel(cluster) log [DBG] : 2.18 scrub starts
Sep 13 17:16:28 str-237 bash[3607700]: debug 2023-09-13T17:16:28.779+0000 7f3a3ddff700 0 log_channel(cluster) log [DBG] : 2.18 scrub ok
Sep 13 17:16:44 str-237 bash[3607700]: debug 2023-09-13T17:16:44.987+0000 7f3a38df5700 0 log_channel(cluster) log [DBG] : 2.aa deep-scrub starts
Sep 13 17:16:45 str-237 bash[3607700]: debug 2023-09-13T17:16:45.979+0000 7f3a38df5700 0 log_channel(cluster) log [DBG] : 2.aa scrub starts
Sep 13 17:16:47 str-237 bash[3607700]: debug 2023-09-13T17:16:47.963+0000 7f3a3cdfd700 0 log_channel(cluster) log [DBG] : 2.aa scrub starts
Sep 13 17:16:51 str-237 bash[3607700]: debug 2023-09-13T17:16:51.983+0000 7f3a3cdfd700 0 log_channel(cluster) log [DBG] : 2.aa scrub starts
Sep 13 17:16:57 str-237 bash[3607700]: debug 2023-09-13T17:16:57.051+0000 7f3a38df5700 0 log_channel(cluster) log [DBG] : 2.aa deep-scrub starts
Sep 13 17:17:00 str-237 bash[3607700]: debug 2023-09-13T17:17:00.067+0000 7f3a38df5700 0 log_channel(cluster) log [DBG] : 2.aa scrub starts
Sep 13 17:17:14 str-237 bash[3607700]: debug 2023-09-13T17:17:14.676+0000 7f3a3cdfd700 0 log_channel(cluster) log [DBG] : 2.aa scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.13.service - Ceph osd.13 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:40 UTC; 9min ago
Main PID: 3535565 (bash)
Tasks: 25 (limit: 618873)
Memory: 21.8M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.13.service
├─3535565 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.13/unit.run
└─3607436 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-13 -e CONTAINER_IMAGE=q>
Sep 13 17:15:56 str-237 bash[3607436]: debug 2023-09-13T17:15:56.434+0000 7f3d941e5700 1 osd.13 pg_epoch: 30207 pg[2.a6( v 30088'5968439 (29926'5960456,30088'5968439] local-lis/les=27609/27610 n=87491 ec=83/83 lis/c=27609/27609 les/c/f>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d919e0700 1 osd.13 pg_epoch: 30208 pg[4.b( empty local-lis/les=16271/16272 n=0 ec=95/95 lis/c=16271/16271 les/c/f=16272/16272/0 sis=30207) [13,37] r=0 lpr=302>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d919e0700 1 osd.13 pg_epoch: 30208 pg[4.b( empty local-lis/les=16271/16272 n=0 ec=95/95 lis/c=16271/16271 les/c/f=16272/16272/0 sis=30208) [13,31,37] r=0 lpr=>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d929e2700 1 osd.13 pg_epoch: 30208 pg[2.9( v 30133'1639172 (29926'1632334,30133'1639172] local-lis/les=28673/28674 n=87414 ec=83/83 lis/c=28673/28673 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d929e2700 1 osd.13 pg_epoch: 30208 pg[2.9( v 30133'1639172 (29926'1632334,30133'1639172] local-lis/les=28673/28674 n=87414 ec=83/83 lis/c=28673/28673 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d919e0700 1 osd.13 pg_epoch: 30208 pg[4.b( empty local-lis/les=16271/16272 n=0 ec=95/95 lis/c=16271/16271 les/c/f=16272/16272/0 sis=30208) [13,31,37] r=0 lpr=>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d969ea700 1 osd.13 pg_epoch: 30208 pg[2.71( v 30111'1083279 (29926'1076458,30111'1083279] local-lis/les=27986/27987 n=87523 ec=83/83 lis/c=27986/27986 les/c/f>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d969ea700 1 osd.13 pg_epoch: 30208 pg[2.71( v 30111'1083279 (29926'1076458,30111'1083279] local-lis/les=27986/27987 n=87523 ec=83/83 lis/c=27986/27986 les/c/f>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d951e7700 1 osd.13 pg_epoch: 30208 pg[2.6c( v 30151'657517 (29926'652156,30151'657517] local-lis/les=28015/28016 n=87254 ec=83/83 lis/c=28015/28015 les/c/f=28>
Sep 13 17:15:57 str-237 bash[3607436]: debug 2023-09-13T17:15:57.434+0000 7f3d951e7700 1 osd.13 pg_epoch: 30208 pg[2.6c( v 30151'657517 (29926'652156,30151'657517] local-lis/les=28015/28016 n=87254 ec=83/83 lis/c=28015/28015 les/c/f=28>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.29.service - Ceph osd.29 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535594 (bash)
Tasks: 22 (limit: 618873)
Memory: 21.5M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.29.service
├─3535594 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.29/unit.run
└─3609290 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-29 -e CONTAINER_IMAGE=q>
Sep 13 17:15:56 str-237 bash[3609290]: debug 2023-09-13T17:15:56.434+0000 7fb43810c700 1 osd.29 pg_epoch: 30207 pg[2.4f( v 30061'1534826 (29926'1528670,30061'1534826] local-lis/les=28205/28206 n=87156 ec=83/83 lis/c=28205/28205 les/c/f>
Sep 13 17:15:56 str-237 bash[3609290]: debug 2023-09-13T17:15:56.434+0000 7fb43b913700 1 osd.29 pg_epoch: 30207 pg[2.d0( v 30018'1485715 (29926'1480423,30018'1485715] local-lis/les=27325/27326 n=87479 ec=83/83 lis/c=27325/27325 les/c/f>
Sep 13 17:15:56 str-237 bash[3609290]: debug 2023-09-13T17:15:56.434+0000 7fb43710a700 1 osd.29 pg_epoch: 30207 pg[4.19( empty local-lis/les=16071/16072 n=0 ec=95/95 lis/c=16071/16071 les/c/f=16072/16072/0 sis=30207) [2,29,33] r=1 lpr=>
Sep 13 17:15:56 str-237 bash[3609290]: debug 2023-09-13T17:15:56.434+0000 7fb43790b700 1 osd.29 pg_epoch: 30207 pg[6.0( v 9249'4955952 (8970'4954873,9249'4955952] local-lis/les=13699/13700 n=574 ec=3863/3863 lis/c=13699/13699 les/c/f=1>
Sep 13 17:15:56 str-237 bash[3609290]: debug 2023-09-13T17:15:56.434+0000 7fb43b913700 1 osd.29 pg_epoch: 30207 pg[2.d0( v 30018'1485715 (29926'1480423,30018'1485715] local-lis/les=27325/27326 n=87479 ec=83/83 lis/c=27325/27325 les/c/f>
Sep 13 17:15:56 str-237 bash[3609290]: debug 2023-09-13T17:15:56.434+0000 7fb43890d700 1 osd.29 pg_epoch: 30207 pg[2.66( v 30054'1224484 (29926'1217622,30054'1224484] local-lis/les=28758/28759 n=87395 ec=83/83 lis/c=28758/28758 les/c/f>
Sep 13 17:15:57 str-237 bash[3609290]: debug 2023-09-13T17:15:57.434+0000 7fb43710a700 1 osd.29 pg_epoch: 30208 pg[4.11( empty local-lis/les=16184/16185 n=0 ec=95/95 lis/c=16184/16184 les/c/f=16185/16185/0 sis=30208) [21,31,29] r=2 lpr>
Sep 13 17:15:57 str-237 bash[3609290]: debug 2023-09-13T17:15:57.434+0000 7fb435106700 1 osd.29 pg_epoch: 30208 pg[2.25( v 30078'786055 (29895'778087,30078'786055] local-lis/les=28504/28505 n=87429 ec=83/83 lis/c=28504/28504 les/c/f=28>
Sep 13 17:15:57 str-237 bash[3609290]: debug 2023-09-13T17:15:57.434+0000 7fb435106700 1 osd.29 pg_epoch: 30208 pg[2.25( v 30078'786055 (29895'778087,30078'786055] local-lis/les=28504/28505 n=87429 ec=83/83 lis/c=28504/28504 les/c/f=28>
Sep 13 17:15:57 str-237 bash[3609290]: debug 2023-09-13T17:15:57.434+0000 7fb43710a700 1 osd.29 pg_epoch: 30208 pg[4.11( empty local-lis/les=16184/16185 n=0 ec=95/95 lis/c=16184/16184 les/c/f=16185/16185/0 sis=30208) [21,31,29] r=2 lpr>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.33.service - Ceph osd.33 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535605 (bash)
Tasks: 28 (limit: 618873)
Memory: 22.6M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.33.service
├─3535605 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.33/unit.run
└─3609233 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-33 -e CONTAINER_IMAGE=q>
Sep 13 17:15:56 str-237 bash[3609233]: debug 2023-09-13T17:15:56.434+0000 7f045614b700 1 osd.33 pg_epoch: 30207 pg[2.fc( v 30110'985573 (29926'979875,30110'985573] local-lis/les=27037/27038 n=87399 ec=83/83 lis/c=27037/27037 les/c/f=27>
Sep 13 17:15:56 str-237 bash[3609233]: debug 2023-09-13T17:15:56.434+0000 7f045594a700 1 osd.33 pg_epoch: 30207 pg[2.7d( v 30088'646523 (29926'640386,30088'646523] local-lis/les=27893/27894 n=87346 ec=83/83 lis/c=27893/27893 les/c/f=27>
Sep 13 17:15:56 str-237 bash[3609233]: debug 2023-09-13T17:15:56.434+0000 7f045594a700 1 osd.33 pg_epoch: 30207 pg[2.7d( v 30088'646523 (29926'640386,30088'646523] local-lis/les=27893/27894 n=87346 ec=83/83 lis/c=27893/27893 les/c/f=27>
Sep 13 17:15:56 str-237 bash[3609233]: debug 2023-09-13T17:15:56.434+0000 7f0459151700 1 osd.33 pg_epoch: 30207 pg[6.1e( v 8983'4514640 (8970'4513567,8983'4514640] local-lis/les=13740/13741 n=577 ec=11565/3863 lis/c=13740/13740 les/c/f>
Sep 13 17:15:56 str-237 bash[3609233]: debug 2023-09-13T17:15:56.434+0000 7f0459151700 1 osd.33 pg_epoch: 30207 pg[6.1e( v 8983'4514640 (8970'4513567,8983'4514640] local-lis/les=13740/13741 n=577 ec=11565/3863 lis/c=13740/13740 les/c/f>
Sep 13 17:15:57 str-237 bash[3609233]: debug 2023-09-13T17:15:57.434+0000 7f045694c700 1 osd.33 pg_epoch: 30208 pg[2.83( v 30097'772009 (29920'765820,30097'772009] local-lis/les=27863/27864 n=87597 ec=83/83 lis/c=27863/27863 les/c/f=27>
Sep 13 17:15:57 str-237 bash[3609233]: debug 2023-09-13T17:15:57.438+0000 7f045694c700 1 osd.33 pg_epoch: 30208 pg[2.83( v 30097'772009 (29920'765820,30097'772009] local-lis/les=27863/27864 n=87597 ec=83/83 lis/c=27863/27863 les/c/f=27>
Sep 13 17:15:57 str-237 bash[3609233]: debug 2023-09-13T17:15:57.438+0000 7f045694c700 1 osd.33 pg_epoch: 30208 pg[2.83( v 30097'772009 (29920'765820,30097'772009] local-lis/les=27863/27864 n=87597 ec=83/83 lis/c=27863/27863 les/c/f=27>
Sep 13 17:15:57 str-237 bash[3609233]: debug 2023-09-13T17:15:57.438+0000 7f0455149700 1 osd.33 pg_epoch: 30208 pg[6.1e( v 8983'4514640 (8970'4513567,8983'4514640] local-lis/les=13740/13741 n=577 ec=11565/3863 lis/c=13740/13740 les/c/f>
Sep 13 17:15:57 str-237 bash[3609233]: debug 2023-09-13T17:15:57.438+0000 7f0455149700 1 osd.33 pg_epoch: 30208 pg[6.1e( v 8983'4514640 (8970'4513567,8983'4514640] local-lis/les=13740/13741 n=577 ec=11565/3863 lis/c=13740/13740 les/c/f>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.5.service - Ceph osd.5 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535660 (bash)
Tasks: 31 (limit: 618873)
Memory: 21.7M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.5.service
├─3535660 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.5/unit.run
└─3607678 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-5 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:57 str-237 bash[3607678]: debug 2023-09-13T17:15:57.434+0000 7febe2c71700 1 osd.5 pg_epoch: 30208 pg[2.79( v 30041'1919586 (29891'1911250,30041'1919586] local-lis/les=27919/27920 n=86631 ec=83/83 lis/c=27919/27919 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607678]: debug 2023-09-13T17:15:57.434+0000 7febdc464700 1 osd.5 pg_epoch: 30208 pg[4.16( empty local-lis/les=16114/16115 n=0 ec=95/95 lis/c=16114/16114 les/c/f=16115/16115/0 sis=30207) [5,17] r=0 lpr=3020>
Sep 13 17:15:57 str-237 bash[3607678]: debug 2023-09-13T17:15:57.434+0000 7febdc464700 1 osd.5 pg_epoch: 30208 pg[4.16( empty local-lis/les=16114/16115 n=0 ec=95/95 lis/c=16114/16114 les/c/f=16115/16115/0 sis=30208) [5,17,31] r=0 lpr=3>
Sep 13 17:15:57 str-237 bash[3607678]: debug 2023-09-13T17:15:57.434+0000 7febdc464700 1 osd.5 pg_epoch: 30208 pg[4.16( empty local-lis/les=16114/16115 n=0 ec=95/95 lis/c=16114/16114 les/c/f=16115/16115/0 sis=30208) [5,17,31] r=0 lpr=3>
Sep 13 17:15:58 str-237 bash[3607678]: debug 2023-09-13T17:15:58.158+0000 7febdcc65700 0 log_channel(cluster) log [DBG] : 2.d5 scrub starts
Sep 13 17:16:13 str-237 bash[3607678]: debug 2023-09-13T17:16:13.290+0000 7febe0c6d700 0 log_channel(cluster) log [DBG] : 2.d5 scrub ok
Sep 13 17:16:25 str-237 bash[3607678]: debug 2023-09-13T17:16:25.374+0000 7febe2c71700 0 log_channel(cluster) log [DBG] : 2.79 scrub starts
Sep 13 17:16:26 str-237 bash[3607678]: debug 2023-09-13T17:16:26.394+0000 7febe2c71700 0 log_channel(cluster) log [DBG] : 2.79 scrub starts
Sep 13 17:16:29 str-237 bash[3607678]: debug 2023-09-13T17:16:29.375+0000 7febdec69700 0 log_channel(cluster) log [DBG] : 2.79 scrub starts
Sep 13 17:16:44 str-237 bash[3607678]: debug 2023-09-13T17:16:44.083+0000 7febe2c71700 0 log_channel(cluster) log [DBG] : 2.79 scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.1.service - Ceph osd.1 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:40 UTC; 9min ago
Main PID: 3535544 (bash)
Tasks: 22 (limit: 618873)
Memory: 21.5M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.1.service
├─3535544 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.1/unit.run
└─3607962 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-1 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbee156700 1 osd.1 pg_epoch: 30208 pg[2.9( v 30133'1639172 (29926'1632334,30133'1639172] local-lis/les=28673/28674 n=87414 ec=83/83 lis/c=28673/28673 les/c/f=2>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbeb951700 1 osd.1 pg_epoch: 30208 pg[2.ee( v 30066'1151992 (29926'1146382,30066'1151992] local-lis/les=27135/27136 n=87710 ec=83/83 lis/c=27135/27135 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbee156700 1 osd.1 pg_epoch: 30208 pg[2.9( v 30133'1639172 (29926'1632334,30133'1639172] local-lis/les=28673/28674 n=87414 ec=83/83 lis/c=28673/28673 les/c/f=2>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbe7148700 1 osd.1 pg_epoch: 30208 pg[2.3f( v 30034'1299321 (29926'1293613,30034'1299321] local-lis/les=28326/28327 n=87974 ec=83/83 lis/c=28326/28326 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbee156700 1 osd.1 pg_epoch: 30208 pg[2.9( v 30133'1639172 (29926'1632334,30133'1639172] local-lis/les=28673/28674 n=87414 ec=83/83 lis/c=28673/28673 les/c/f=2>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbe7148700 1 osd.1 pg_epoch: 30208 pg[2.3f( v 30034'1299321 (29926'1293613,30034'1299321] local-lis/les=28326/28327 n=87974 ec=83/83 lis/c=28326/28326 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbeb951700 1 osd.1 pg_epoch: 30208 pg[2.ee( v 30066'1151992 (29926'1146382,30066'1151992] local-lis/les=27135/27136 n=87710 ec=83/83 lis/c=27135/27135 les/c/f=>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbeb150700 1 osd.1 pg_epoch: 30208 pg[4.7( empty local-lis/les=16306/16307 n=0 ec=95/95 lis/c=16306/16306 les/c/f=16307/16307/0 sis=30207) [1,39] r=0 lpr=30207>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbeb150700 1 osd.1 pg_epoch: 30208 pg[4.7( empty local-lis/les=16306/16307 n=0 ec=95/95 lis/c=16306/16306 les/c/f=16307/16307/0 sis=30208) [1,39,31] r=0 lpr=30>
Sep 13 17:15:57 str-237 bash[3607962]: debug 2023-09-13T17:15:57.438+0000 7fcbeb150700 1 osd.1 pg_epoch: 30208 pg[4.7( empty local-lis/les=16306/16307 n=0 ec=95/95 lis/c=16306/16306 les/c/f=16307/16307/0 sis=30208) [1,39,31] r=0 lpr=30>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.9.service - Ceph osd.9 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535667 (bash)
Tasks: 24 (limit: 618873)
Memory: 21.6M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.9.service
├─3535667 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.9/unit.run
└─3608989 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-9 -e CONTAINER_IMAGE=qu>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805c727700 1 osd.9 pg_epoch: 30208 pg[2.c9( v 30013'632450 (29926'624734,30013'632450] local-lis/les=27380/27381 n=87259 ec=83/83 lis/c=27380/27380 les/c/f=273>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805d729700 1 osd.9 pg_epoch: 30208 pg[5.f( v 29875'247 (11251'199,29875'247] local-lis/les=16957/16958 n=23 ec=97/97 lis/c=16957/16957 les/c/f=16958/16958/0 si>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805cf28700 1 osd.9 pg_epoch: 30208 pg[2.50( v 30111'997422 (29926'991001,30111'997422] local-lis/les=28201/28202 n=87898 ec=83/83 lis/c=28201/28201 les/c/f=282>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805bf26700 1 osd.9 pg_epoch: 30208 pg[4.12( empty local-lis/les=16180/16181 n=0 ec=95/95 lis/c=16180/16180 les/c/f=16181/16181/0 sis=30208) [31,9,27] r=1 lpr=3>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805bf26700 1 osd.9 pg_epoch: 30208 pg[4.12( empty local-lis/les=16180/16181 n=0 ec=95/95 lis/c=16180/16180 les/c/f=16181/16181/0 sis=30208) [31,9,27] r=1 lpr=3>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805cf28700 1 osd.9 pg_epoch: 30208 pg[2.50( v 30111'997422 (29926'991001,30111'997422] local-lis/les=28201/28202 n=87898 ec=83/83 lis/c=28201/28201 les/c/f=282>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f805d729700 1 osd.9 pg_epoch: 30208 pg[5.f( v 29875'247 (11251'199,29875'247] local-lis/les=16957/16958 n=23 ec=97/97 lis/c=16957/16957 les/c/f=16958/16958/0 si>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f8060f30700 1 osd.9 pg_epoch: 30208 pg[5.18( v 29873'270 (11251'215,29873'270] local-lis/les=17253/17254 n=21 ec=97/97 lis/c=17253/17253 les/c/f=17254/17254/0 s>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f8060f30700 1 osd.9 pg_epoch: 30208 pg[5.18( v 29873'270 (11251'215,29873'270] local-lis/les=17253/17254 n=21 ec=97/97 lis/c=17253/17253 les/c/f=17254/17254/0 s>
Sep 13 17:15:57 str-237 bash[3608989]: debug 2023-09-13T17:15:57.438+0000 7f8060f30700 1 osd.9 pg_epoch: 30208 pg[5.18( v 29873'270 (11251'215,29873'270] local-lis/les=17253/17254 n=21 ec=97/97 lis/c=17253/17253 les/c/f=17254/17254/0 s>
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.45.service - Ceph osd.45 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535652 (bash)
Tasks: 28 (limit: 618873)
Memory: 22.9M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.45.service
├─3535652 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.45/unit.run
└─3610146 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-45 -e CONTAINER_IMAGE=q>
Sep 13 17:15:57 str-237 bash[3610146]: debug 2023-09-13T17:15:57.434+0000 7fdc7fc0c700 1 osd.45 pg_epoch: 30208 pg[4.13( empty local-lis/les=17249/17250 n=0 ec=95/95 lis/c=17249/17249 les/c/f=17250/17250/0 sis=30208) [45,39,31] r=0 lpr>
Sep 13 17:15:57 str-237 bash[3610146]: debug 2023-09-13T17:15:57.434+0000 7fdc8040d700 1 osd.45 pg_epoch: 30208 pg[3.12( empty local-lis/les=16835/16836 n=0 ec=94/94 lis/c=16835/16835 les/c/f=16836/16836/0 sis=30208) [45,7,31] r=0 lpr=>
Sep 13 17:15:57 str-237 bash[3610146]: debug 2023-09-13T17:15:57.434+0000 7fdc7dc08700 1 osd.45 pg_epoch: 30208 pg[2.d7( v 30116'829911 (29926'823699,30116'829911] local-lis/les=27286/27287 n=87491 ec=83/83 lis/c=27286/27286 les/c/f=27>
Sep 13 17:16:15 str-237 bash[3610146]: debug 2023-09-13T17:16:15.530+0000 7fdc8040d700 0 log_channel(cluster) log [DBG] : 2.1a scrub starts
Sep 13 17:16:17 str-237 bash[3610146]: debug 2023-09-13T17:16:17.526+0000 7fdc8040d700 0 log_channel(cluster) log [DBG] : 2.1a scrub starts
Sep 13 17:16:20 str-237 bash[3610146]: debug 2023-09-13T17:16:20.538+0000 7fdc7c405700 0 log_channel(cluster) log [DBG] : 2.1a scrub starts
Sep 13 17:16:23 str-237 bash[3610146]: debug 2023-09-13T17:16:23.582+0000 7fdc8040d700 0 log_channel(cluster) log [DBG] : 2.1a scrub starts
Sep 13 17:16:26 str-237 bash[3610146]: debug 2023-09-13T17:16:26.614+0000 7fdc8040d700 0 log_channel(cluster) log [DBG] : 2.1a scrub starts
Sep 13 17:16:30 str-237 bash[3610146]: debug 2023-09-13T17:16:30.627+0000 7fdc7c405700 0 log_channel(cluster) log [DBG] : 2.1a scrub starts
Sep 13 17:16:45 str-237 bash[3610146]: debug 2023-09-13T17:16:45.399+0000 7fdc8040d700 0 log_channel(cluster) log [DBG] : 2.1a scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.25.service - Ceph osd.25 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535587 (bash)
Tasks: 29 (limit: 618873)
Memory: 22.2M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.25.service
├─3535587 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.25/unit.run
└─3609494 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-25 -e CONTAINER_IMAGE=q>
Sep 13 17:15:57 str-237 bash[3609494]: debug 2023-09-13T17:15:57.438+0000 7f435505e700 1 osd.25 pg_epoch: 30208 pg[2.6d( v 30113'1253590 (29926'1247125,30113'1253590] local-lis/les=28753/28754 n=86815 ec=83/83 lis/c=28753/28753 les/c/f>
Sep 13 17:15:57 str-237 bash[3609494]: debug 2023-09-13T17:15:57.438+0000 7f435505e700 1 osd.25 pg_epoch: 30208 pg[2.6d( v 30113'1253590 (29926'1247125,30113'1253590] local-lis/les=28753/28754 n=86815 ec=83/83 lis/c=28753/28753 les/c/f>
Sep 13 17:15:57 str-237 bash[3609494]: debug 2023-09-13T17:15:57.438+0000 7f4350855700 1 osd.25 pg_epoch: 30208 pg[2.4e( v 30062'877616 (29895'870730,30062'877616] local-lis/les=28215/28216 n=86923 ec=83/83 lis/c=28215/28215 les/c/f=28>
Sep 13 17:16:13 str-237 bash[3609494]: debug 2023-09-13T17:16:13.350+0000 7f4356861700 0 log_channel(cluster) log [DBG] : 2.52 scrub starts
Sep 13 17:16:27 str-237 bash[3609494]: debug 2023-09-13T17:16:27.979+0000 7f4356861700 0 log_channel(cluster) log [DBG] : 2.52 scrub ok
Sep 13 17:16:29 str-237 bash[3609494]: debug 2023-09-13T17:16:29.247+0000 7f435505e700 0 log_channel(cluster) log [DBG] : 5.d scrub starts
Sep 13 17:16:30 str-237 bash[3609494]: debug 2023-09-13T17:16:30.215+0000 7f435505e700 0 log_channel(cluster) log [DBG] : 5.d scrub starts
Sep 13 17:16:31 str-237 bash[3609494]: debug 2023-09-13T17:16:31.219+0000 7f435505e700 0 log_channel(cluster) log [DBG] : 5.d scrub starts
Sep 13 17:19:00 str-237 bash[3609494]: debug 2023-09-13T17:19:00.698+0000 7f4351056700 0 log_channel(cluster) log [DBG] : 5.d scrub starts
Sep 13 17:19:00 str-237 bash[3609494]: debug 2023-09-13T17:19:00.710+0000 7f435505e700 0 log_channel(cluster) log [DBG] : 5.d scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.41.service - Ceph osd.41 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535633 (bash)
Tasks: 27 (limit: 618873)
Memory: 22.2M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.41.service
├─3535633 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.41/unit.run
└─3609876 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-41 -e CONTAINER_IMAGE=q>
Sep 13 17:18:56 str-237 bash[3609876]: debug 2023-09-13T17:18:56.098+0000 7f4e3dece700 0 log_channel(cluster) log [DBG] : 2.16 scrub starts
Sep 13 17:18:58 str-237 bash[3609876]: debug 2023-09-13T17:18:58.186+0000 7f4e3b6c9700 0 log_channel(cluster) log [DBG] : 3.1b scrub starts
Sep 13 17:19:07 str-237 bash[3609876]: debug 2023-09-13T17:19:07.246+0000 7f4e3dece700 0 log_channel(cluster) log [DBG] : 2.16 scrub starts
Sep 13 17:19:13 str-237 bash[3609876]: debug 2023-09-13T17:19:13.406+0000 7f4e3b6c9700 0 log_channel(cluster) log [DBG] : 3.1b scrub starts
Sep 13 17:19:16 str-237 bash[3609876]: debug 2023-09-13T17:19:16.478+0000 7f4e3dece700 0 log_channel(cluster) log [DBG] : 2.16 scrub starts
Sep 13 17:19:17 str-237 bash[3609876]: debug 2023-09-13T17:19:17.450+0000 7f4e3b6c9700 0 log_channel(cluster) log [DBG] : 3.1b scrub starts
Sep 13 17:19:18 str-237 bash[3609876]: debug 2023-09-13T17:19:18.410+0000 7f4e3dece700 0 log_channel(cluster) log [DBG] : 2.16 scrub starts
Sep 13 17:19:32 str-237 bash[3609876]: debug 2023-09-13T17:19:32.983+0000 7f4e3dece700 0 log_channel(cluster) log [DBG] : 2.16 scrub ok
Sep 13 17:19:34 str-237 bash[3609876]: debug 2023-09-13T17:19:34.195+0000 7f4e3f6d1700 0 log_channel(cluster) log [DBG] : 3.1b scrub starts
Sep 13 17:19:34 str-237 bash[3609876]: debug 2023-09-13T17:19:34.203+0000 7f4e3f6d1700 0 log_channel(cluster) log [DBG] : 3.1b scrub ok
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.17.service - Ceph osd.17 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:41 UTC; 9min ago
Main PID: 3535577 (bash)
Tasks: 30 (limit: 618873)
Memory: 22.0M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.17.service
├─3535577 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/osd.17/unit.run
└─3607651 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-osd-17 -e CONTAINER_IMAGE=q>
Sep 13 17:16:26 str-237 bash[3607651]: debug 2023-09-13T17:16:26.470+0000 7fceeecae700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:29 str-237 bash[3607651]: debug 2023-09-13T17:16:29.479+0000 7fceeecae700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:30 str-237 bash[3607651]: debug 2023-09-13T17:16:30.439+0000 7fcef2cb6700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:31 str-237 bash[3607651]: debug 2023-09-13T17:16:31.431+0000 7fceeecae700 0 log_channel(cluster) log [DBG] : 2.50 deep-scrub starts
Sep 13 17:16:32 str-237 bash[3607651]: debug 2023-09-13T17:16:32.423+0000 7fcef2cb6700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:37 str-237 bash[3607651]: debug 2023-09-13T17:16:37.419+0000 7fcef2cb6700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:38 str-237 bash[3607651]: debug 2023-09-13T17:16:38.387+0000 7fceeecae700 0 log_channel(cluster) log [DBG] : 2.50 deep-scrub starts
Sep 13 17:16:40 str-237 bash[3607651]: debug 2023-09-13T17:16:40.395+0000 7fcef2cb6700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:44 str-237 bash[3607651]: debug 2023-09-13T17:16:44.303+0000 7fceeecae700 0 log_channel(cluster) log [DBG] : 2.50 scrub starts
Sep 13 17:16:59 str-237 bash[3607651]: debug 2023-09-13T17:16:59.059+0000 7fcef2cb6700 0 log_channel(cluster) log [DBG] : 2.50 scrub ok
Ok that’s all 24 osds (12+12) looking happy check the last mon
● ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@mon.str-237.service - Ceph mon.str-237 for 75b74760-4e61-11ed-b975-0b6dbf7a8a34
Loaded: loaded (/etc/systemd/system/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-13 17:13:40 UTC; 9min ago
Main PID: 3535521 (bash)
Tasks: 30 (limit: 618873)
Memory: 19.6M
CGroup: /system.slice/system-ceph\x2d75b74760\x2d4e61\x2d11ed\x2db975\x2d0b6dbf7a8a34.slice/ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@mon.str-237.service
├─3535521 /bin/bash /var/lib/ceph/75b74760-4e61-11ed-b975-0b6dbf7a8a34/mon.str-237/unit.run
└─3535806 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34-mon-str-237 -e CONTAINER_IM>
Sep 13 17:22:46 str-237 bash[3535806]: cluster 2023-09-13T17:22:44.964247+0000 mgr.str-238.aujped (mgr.15644118) 238 : cluster [DBG] pgmap v238: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Sep 13 17:22:46 str-237 bash[3535806]: debug 2023-09-13T17:22:46.855+0000 7f72b68c0700 1 mon.str-237@1(peon).osd e30210 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
Sep 13 17:22:48 str-237 bash[3535806]: cluster 2023-09-13T17:22:46.965032+0000 mgr.str-238.aujped (mgr.15644118) 239 : cluster [DBG] pgmap v239: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Sep 13 17:22:50 str-237 bash[3535806]: cluster 2023-09-13T17:22:48.966814+0000 mgr.str-238.aujped (mgr.15644118) 240 : cluster [DBG] pgmap v240: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Sep 13 17:22:51 str-237 bash[3535806]: cluster 2023-09-13T17:22:50.967976+0000 mgr.str-238.aujped (mgr.15644118) 241 : cluster [DBG] pgmap v241: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Sep 13 17:22:51 str-237 bash[3535806]: debug 2023-09-13T17:22:51.855+0000 7f72b68c0700 1 mon.str-237@1(peon).osd e30210 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
Sep 13 17:22:54 str-237 bash[3535806]: cluster 2023-09-13T17:22:52.969416+0000 mgr.str-238.aujped (mgr.15644118) 242 : cluster [DBG] pgmap v242: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Sep 13 17:22:56 str-237 bash[3535806]: cluster 2023-09-13T17:22:54.970788+0000 mgr.str-238.aujped (mgr.15644118) 243 : cluster [DBG] pgmap v243: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Sep 13 17:22:56 str-237 bash[3535806]: debug 2023-09-13T17:22:56.855+0000 7f72b68c0700 1 mon.str-237@1(peon).osd e30210 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408
Sep 13 17:22:58 str-237 bash[3535806]: cluster 2023-09-13T17:22:56.971567+0000 mgr.str-238.aujped (mgr.15644118) 244 : cluster [DBG] pgmap v244: 385 pgs: 1 active+clean+scrubbing+deep, 384 active+clean; 24 TiB data, 62 TiB used, 106 TiB>
Great now lets unset each flag one by one watching the status between each one in case we missed something
root@str-237:~# ceph osd unset pause
pauserd,pausewr is unset
root@str-237:~# ceph -s
cluster:
id: 75b74760-4e61-11ed-b975-0b6dbf7a8a34
health: HEALTH_WARN
1 hosts fail cephadm check
nodown,noout,nobackfill,norebalance,norecover flag(s) set
services:
mon: 2 daemons, quorum str-238,str-237 (age 8m)
mgr: str-238.aujped(active, since 8m), standbys: str-237.aicbzp
osd: 24 osds: 24 up (since 7m), 24 in (since 11M)
flags nodown,noout,nobackfill,norebalance,norecover
data:
pools: 6 pools, 385 pgs
objects: 22.34M objects, 24 TiB
usage: 62 TiB used, 106 TiB / 168 TiB avail
pgs: 384 active+clean
1 active+clean+scrubbing+deep
io:
client: 106 MiB/s rd, 204 B/s wr, 38 op/s rd, 2 op/s wr
root@str-237:~# ceph -s
cluster:
id: 75b74760-4e61-11ed-b975-0b6dbf7a8a34
health: HEALTH_WARN
1 hosts fail cephadm check
nodown,noout,nobackfill,norebalance,norecover flag(s) set
services:
mon: 2 daemons, quorum str-238,str-237 (age 8m)
mgr: str-238.aujped(active, since 8m), standbys: str-237.aicbzp
osd: 24 osds: 24 up (since 7m), 24 in (since 11M)
flags nodown,noout,nobackfill,norebalance,norecover
data:
pools: 6 pools, 385 pgs
objects: 22.34M objects, 24 TiB
usage: 62 TiB used, 106 TiB / 168 TiB avail
pgs: 384 active+clean
1 active+clean+scrubbing+deep
io:
client: 294 MiB/s rd, 36 op/s rd, 0 op/s wr
root@str-237:~# ceph osd unset nodown
nodown is unset
root@str-237:~# watch ceph -s
root@str-237:~# ceph osd unset norebalance
norebalance is unset
root@str-237:~# watch ceph -s
root@str-237:~# ceph osd unset norecover
norecover is unset
root@str-237:~# watch ceph -s
root@str-237:~# ceph osd unset nobackfill
nobackfill is unset
root@str-237:~# watch ceph -s
root@str-237:~# ceph osd unset noout
noout is unset
root@str-237:~# watch ceph -s
root@str-237:~# watch ceph -s
Every 2.0s: ceph -s str-237: Wed Sep 13 18:10:27 2023
cluster:
id: 75b74760-4e61-11ed-b975-0b6dbf7a8a34
health: HEALTH_WARN
1 hosts fail cephadm check
services:
mon: 2 daemons, quorum str-238,str-237 (age 55m)
mgr: str-238.aujped(active, since 55m), standbys: str-237.aicbzp
osd: 24 osds: 24 up (since 54m), 24 in (since 11M)
data:
pools: 6 pools, 385 pgs
objects: 22.34M objects, 24 TiB
usage: 62 TiB used, 106 TiB / 168 TiB avail
pgs: 385 active+clean
back up!
ok I could has gone faster but better safe than sorry
From: David Power <david@define-technology.com>
Sent: Wednesday, September 13, 2023 5:36 PM
To: Antony Cleave <antony@define-technology.com>
Subject: Few notes on Ceph stop at Redmond
root@str-237:~# ceph status
cluster:
id: 75b74760-4e61-11ed-b975-0b6dbf7a8a34
health: HEALTH_WARN
1 hosts fail cephadm check
services:
mon: 2 daemons, quorum str-238,str-237 (age 9w)
mgr: str-238.aujped(active, since 4M), standbys: str-237.aicbzp
osd: 24 osds: 24 up (since 9w), 24 in (since 11M)
data:
pools: 6 pools, 385 pgs
objects: 22.34M objects, 24 TiB
usage: 62 TiB used, 106 TiB / 168 TiB avail
pgs: 385 active+clean
root@str-237:~# ceph osd set noout
noout is set
root@str-237:~# ceph osd set nobackfill
nobackfill is set
root@str-237:~# ceph osd set norecover
norecover is set
root@str-237:~# ceph osd set norebalance
norebalance is set
root@str-237:~# ceph osd set nodown
nodown is set
root@str-237:~# ceph osd set pause
pauserd,pausewr is set
root@str-237:~# ceph status
cluster:
id: 75b74760-4e61-11ed-b975-0b6dbf7a8a34
health: HEALTH_WARN
1 hosts fail cephadm check
pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set
services:
mon: 2 daemons, quorum str-238,str-237 (age 9w)
mgr: str-238.aujped(active, since 4M), standbys: str-237.aicbzp
osd: 24 osds: 24 up (since 9w), 24 in (since 11M)
flags pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover
data:
pools: 6 pools, 385 pgs
objects: 22.34M objects, 24 TiB
usage: 62 TiB used, 106 TiB / 168 TiB avail
pgs: 385 active+clean
shut down the OSDS
root@str-237:~# systemctl stop ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd.*
root@str-237:~#
root@str-238:~# systemctl stop ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34@osd*
root@str-238:~#
root@str-237:~# systemctl stop ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34*
root@str-237:~#
root@str-238:~# systemctl stop ceph-75b74760-4e61-11ed-b975-0b6dbf7a8a34*
root@str-238:~#