rarias/jungle
2
0
Fork 3
Code Issues 57 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

Ceph OSDs not starting due condition not met #76

New Issue
Open
opened 2024-08-28 12:58:50 +02:00 by rarias · 4 comments
rarias commented 2024-08-28 12:58:50 +02:00
Owner
Copy Link

There is a problem enabling some OSDs:

Aug 28 11:46:57 lake2 systemd[1]: Ceph OSD daemon 5 was skipped because of an unmet condition check (ConditionPathExists=/var/lib/ceph/osd/ceph-5/keyring).

The ceph-volume service has only started OSD 4:

lake2% sudo systemctl status ceph-volume.service
○ ceph-volume.service - Ceph Volume activation
     Loaded: loaded (/etc/systemd/system/ceph-volume.service; enabled; preset: enabled)
     Active: inactive (dead) since Wed 2024-08-28 11:46:54 CEST; 1h 9min ago
   Duration: 1.505s
   Main PID: 1251 (code=exited, status=0/SUCCESS)
        CPU: 744ms

Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/zvb73z94sbh5js93yliqn18bzmzlbayk-ceph-18.2.1/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-b6b9047d-8696-4>
Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/ln -snf /dev/ceph-b6b9047d-8696-43ca-bbd3-9be42aac3151/osd-block-8f3a21b8-e9b0>
Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block
Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/chown -R ceph:ceph /dev/dm-0
Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Aug 28 11:46:56 lake2 sh[1253]: --> ceph-volume lvm activate successful for osd ID: 4
Aug 28 11:46:53 lake2 systemd[1]: Started Ceph Volume activation.
Aug 28 11:46:54 lake2 systemd[1]: ceph-volume.service: Deactivated successfully.
Aug 28 11:46:57 lake2 systemd[1]: /etc/systemd/system/ceph-volume.service:4: Unknown key name 'Type' in section 'Unit', ignoring.
Aug 28 11:46:57 lake2 systemd[1]: /etc/systemd/system/ceph-volume.service:13: Unit uses KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service

WTF there are disk missing:

lake2% ls /dev/nvme*
/dev/nvme0  /dev/nvme0n1
There is a problem enabling some OSDs: ``` Aug 28 11:46:57 lake2 systemd[1]: Ceph OSD daemon 5 was skipped because of an unmet condition check (ConditionPathExists=/var/lib/ceph/osd/ceph-5/keyring). ``` The ceph-volume service has only started OSD 4: ``` lake2% sudo systemctl status ceph-volume.service ○ ceph-volume.service - Ceph Volume activation Loaded: loaded (/etc/systemd/system/ceph-volume.service; enabled; preset: enabled) Active: inactive (dead) since Wed 2024-08-28 11:46:54 CEST; 1h 9min ago Duration: 1.505s Main PID: 1251 (code=exited, status=0/SUCCESS) CPU: 744ms Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/zvb73z94sbh5js93yliqn18bzmzlbayk-ceph-18.2.1/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-b6b9047d-8696-4> Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/ln -snf /dev/ceph-b6b9047d-8696-43ca-bbd3-9be42aac3151/osd-block-8f3a21b8-e9b0> Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/chown -R ceph:ceph /dev/dm-0 Aug 28 11:46:56 lake2 sh[1253]: Running command: /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Aug 28 11:46:56 lake2 sh[1253]: --> ceph-volume lvm activate successful for osd ID: 4 Aug 28 11:46:53 lake2 systemd[1]: Started Ceph Volume activation. Aug 28 11:46:54 lake2 systemd[1]: ceph-volume.service: Deactivated successfully. Aug 28 11:46:57 lake2 systemd[1]: /etc/systemd/system/ceph-volume.service:4: Unknown key name 'Type' in section 'Unit', ignoring. Aug 28 11:46:57 lake2 systemd[1]: /etc/systemd/system/ceph-volume.service:13: Unit uses KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service ``` WTF there are disk missing: ``` lake2% ls /dev/nvme* /dev/nvme0 /dev/nvme0n1
rarias added the hw label 2024-08-28 13:04:04 +02:00
rarias commented 2024-08-28 13:08:50 +02:00
Author
Owner
Copy Link

Let's see if rebooting makes the disk appear again.

Let's see if rebooting makes the disk appear again.
rarias commented 2024-08-28 13:14:34 +02:00
Author
Owner
Copy Link

Right...

lake2% ls /dev/nvme*
/dev/nvme0  /dev/nvme0n1  /dev/nvme1  /dev/nvme1n1  /dev/nvme2  /dev/nvme2n1  /dev/nvme3  /dev/nvme3n1

Now ceph is working fine. Closing.

Right... ``` lake2% ls /dev/nvme* /dev/nvme0 /dev/nvme0n1 /dev/nvme1 /dev/nvme1n1 /dev/nvme2 /dev/nvme2n1 /dev/nvme3 /dev/nvme3n1 ``` Now ceph is working fine. Closing.
rarias reopened this issue 2026-01-28 17:24:42 +01:00
rarias commented 2026-01-28 17:26:04 +01:00
Author
Owner
Copy Link

This is happening again. There is a serious problem during boot that causes the BMC or the BIOS to stop recognizing 3 of the 4 disks used for storage. Given that the BMC is known to be erratic, is probably the culprit.

lake2$ ls /dev/nvme0*
/dev/nvme0  /dev/nvme0n1

Rebooting for now.

This is happening again. There is a serious problem during boot that causes the BMC or the BIOS to stop recognizing 3 of the 4 disks used for storage. Given that the BMC is known to be erratic, is probably the culprit. ``` lake2$ ls /dev/nvme0* /dev/nvme0 /dev/nvme0n1 ``` Rebooting for now.
rarias commented 2026-01-28 17:42:35 +01:00
Author
Owner
Copy Link

Now we have all disks:

lake2% sudo dmesg | grep nvme
[    8.128430] nvme nvme0: pci function 0000:83:00.0
[    8.128707] nvme nvme1: pci function 0000:84:00.0
[    8.128861] nvme nvme3: pci function 0000:86:00.0
[    8.129009] nvme nvme2: pci function 0000:85:00.0
[    8.138461] nvme nvme2: 31/0/0 default/read/poll queues
[    8.138511] nvme nvme1: 31/0/0 default/read/poll queues
[    8.145106] nvme nvme0: 31/0/0 default/read/poll queues
[    8.149120] nvme nvme3: 31/0/0 default/read/poll queues
[   17.083923] nvme nvme1: using unchecked data buffer
Now we have all disks: ``` lake2% sudo dmesg | grep nvme [ 8.128430] nvme nvme0: pci function 0000:83:00.0 [ 8.128707] nvme nvme1: pci function 0000:84:00.0 [ 8.128861] nvme nvme3: pci function 0000:86:00.0 [ 8.129009] nvme nvme2: pci function 0000:85:00.0 [ 8.138461] nvme nvme2: 31/0/0 default/read/poll queues [ 8.138511] nvme nvme1: 31/0/0 default/read/poll queues [ 8.145106] nvme nvme0: 31/0/0 default/read/poll queues [ 8.149120] nvme nvme3: 31/0/0 default/read/poll queues [ 17.083923] nvme nvme1: using unchecked data buffer ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label hw
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon varcila rarias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#76
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?