feat(rook-ceph): re-add neptune's OSD on the PM983 #605

2026-08-25T23:45:02Z

claude-bot commented

2026-08-25 23:45:02 +00:00

neptune is converted and back in the cluster, so k8s-3 returns to the storage fleet on its new PM983. The storage fleet is whole again and all three nodes name their OSD by /dev/disk/by-id.

Verified on the machine before writing it:

$ talosctl -n 10.2.40.138 ls /dev/disk/by-id | grep -i samsung
nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176

Drive health, read from the Proxmox host before the wipe: 3% used, 80.7 TB written, no media errors, no critical warnings.

One thing to do before merging

The old osd.1 is still in the CRUSH map as down/in, holding 335 GiB of references to data that no longer exists — it lived on the Crucial that is now EPHEMERAL, and that drive was wiped during the conversion. While it stays in, PGs mapped to it remain degraded and will not remap.

It needs purging, which I could not run — the command was blocked by a permission classifier:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd purge 1 --yes-i-really-mean-it

Merging this without the purge still works — Rook creates a replacement OSD on the PM983 — but the cluster keeps a phantom osd.1 alongside it and the degraded count will not fully clear. Cleaner to purge first, then merge.

Comment change

The comment generalises from "k8s-1 and k8s-2" to all three nodes and records neptune serials. k8s-3 is the sharpest argument for by-id addressing of the three:

Talos boot 1: nvme0n1 = Kingston  (system disk)
Talos boot 2: nvme0n1 = Crucial   (EPHEMERAL)

Two consecutive boots of the same machine, so a kernel-name reference had no stable meaning there at all.

Expect on merge

Rook provisions the OSD and backfills onto the new drive — roughly 30 minutes at the rates saturn saw. That takes the cluster back to three OSDs across three zones and clears OSD count 2 < osd_pool_default_size 3.

Still open after this

cephfs-metadata back to size 3, .mgr to 3/2, and osd_mclock_profile back to balanced — all deferred until the fleet is whole and backfill has settled.

neptune is converted and back in the cluster, so `k8s-3` returns to the storage fleet on its new PM983. The storage fleet is whole again and all three nodes name their OSD by `/dev/disk/by-id`. Verified on the machine before writing it: ``` $ talosctl -n 10.2.40.138 ls /dev/disk/by-id | grep -i samsung nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176 ``` Drive health, read from the Proxmox host before the wipe: **3% used**, 80.7 TB written, no media errors, no critical warnings. ## One thing to do before merging The old `osd.1` is still in the CRUSH map as `down`/`in`, holding 335 GiB of references to data that no longer exists — it lived on the Crucial that is now `EPHEMERAL`, and that drive was wiped during the conversion. While it stays `in`, PGs mapped to it remain degraded and will not remap. It needs purging, which I could not run — the command was blocked by a permission classifier: ``` kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd purge 1 --yes-i-really-mean-it ``` Merging this without the purge still works — Rook creates a replacement OSD on the PM983 — but the cluster keeps a phantom `osd.1` alongside it and the degraded count will not fully clear. Cleaner to purge first, then merge. ## Comment change The comment generalises from "k8s-1 and k8s-2" to all three nodes and records neptune serials. k8s-3 is the sharpest argument for by-id addressing of the three: ``` Talos boot 1: nvme0n1 = Kingston (system disk) Talos boot 2: nvme0n1 = Crucial (EPHEMERAL) ``` Two consecutive boots of the same machine, so a kernel-name reference had no stable meaning there at all. ## Expect on merge Rook provisions the OSD and backfills onto the new drive — roughly 30 minutes at the rates saturn saw. That takes the cluster back to three OSDs across three zones and clears `OSD count 2 < osd_pool_default_size 3`. ## Still open after this `cephfs-metadata` back to size 3, `.mgr` to 3/2, and `osd_mclock_profile` back to `balanced` — all deferred until the fleet is whole and backfill has settled.

claude-bot added 1 commit 2026-08-25 23:45:02 +00:00

feat(rook-ceph): re-add neptune's OSD on the PM983

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

be0f891c73

neptune is converted and back in the cluster, so k8s-3 returns to the storage
fleet on its new PM983. All three nodes now name their OSD by /dev/disk/by-id,
and the storage fleet is whole again.

The comment generalises to all three nodes and records neptune's serials.
k8s-3 is the clearest evidence for by-id addressing: it enumerated the Kingston
as nvme0n1 on one boot and the Crucial as nvme0n1 on the very next one, so a
kernel-name reference had no stable meaning on that machine at all.

The old osd.1 lived on the Crucial that is now EPHEMERAL. That drive was wiped
during the conversion, so the OSD is unrecoverable and is purged from the CRUSH
map separately; this change only creates its replacement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 0f68a1b344 into master

2026-08-25 23:45:52 +00:00

k.renner referenced this issue from a commit

2026-08-25 23:45:53 +00:00

Merge pull request 'feat(rook-ceph): re-add neptune's OSD on the PM983' (#605) from ceph-readd-neptune into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#605