feat(rook-ceph): restore replicapool and cephfs-data0 to size 3 #610

2026-08-26T00:13:48Z

claude-bot commented

2026-08-26 00:13:48 +00:00

Follow-up to #609, which restored cephfs-metadata. These two pools were left out of that PR pending a decision, and the decision is to restore them.

pool	was	now
`replicapool`	2 / min_size 1	3
`cephfs-data0`	2 / min_size 1	3

Why they were at 2

Not the conversion — this is #434. With three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every drive, so per-drive write volume equalled total cluster write volume and all three wore at an identical rate:

measured 2026-08-19: 80/82/87% endurance used, ~0.13 points/day
projected exhaustion: Nov 2026 - Jan 2027, all three together

Replication is no defence against that, because the wear was correlated by construction rather than independent. Size 2 cut per-drive write volume by a third to desynchronise it, accepting that a second drive failure during a degraded window loses data.

Why that no longer applies

All three OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance with a far higher write budget and independent wear histories:

S437NX0M400426  jupiter  3% used
S437NX0M403161  saturn   2% used
S437NF0M501176  neptune  3% used

The correlated wear-out failure mode the size-2 trade was bought to avoid does not exist on this hardware, so the pools return to the durability the cluster is sized for. This makes #434 resolvable.

min_size

Both min_size: "1" overrides are dropped. They existed only because Ceph does not lower min_size when size is reduced — leaving them would have made the pools 2/2, where any single OSD down blocks IO for its PGs. At size 3 the default min_size of 2 is correct.

Left alone

cephfs-ci stays at size 2 — throwaway Woodpecker CI workspaces, at 2 rather than 1 only because an unreplicated pool makes ceph osd ok-to-stop refuse for every OSD. Unrelated to #434.
objectstore metadataPool is size 2 with no rationale recorded and an erasure-coded 2+1 dataPool. Flagging rather than changing, since I could not establish whether that 2 was deliberate.

Expect on merge

Re-replication of roughly 280 GiB — comparable to the backfill that just completed, so budget similar time and watch ceph -s rather than merging and walking away. The cluster is currently 217 PGs active+clean with all three OSDs up.

Follow-up to #609, which restored `cephfs-metadata`. These two pools were left out of that PR pending a decision, and the decision is to restore them. | pool | was | now | | --- | --- | --- | | `replicapool` | 2 / min_size 1 | 3 | | `cephfs-data0` | 2 / min_size 1 | 3 | ## Why they were at 2 Not the conversion — this is #434. With three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every drive, so per-drive write volume equalled total cluster write volume and all three wore at an identical rate: ``` measured 2026-08-19: 80/82/87% endurance used, ~0.13 points/day projected exhaustion: Nov 2026 - Jan 2027, all three together ``` Replication is no defence against that, because the wear was correlated by construction rather than independent. Size 2 cut per-drive write volume by a third to desynchronise it, accepting that a second drive failure during a degraded window loses data. ## Why that no longer applies All three OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance with a far higher write budget and independent wear histories: ``` S437NX0M400426 jupiter 3% used S437NX0M403161 saturn 2% used S437NF0M501176 neptune 3% used ``` The correlated wear-out failure mode the size-2 trade was bought to avoid does not exist on this hardware, so the pools return to the durability the cluster is sized for. This makes #434 resolvable. ## min_size Both `min_size: "1"` overrides are dropped. They existed only because Ceph does not lower `min_size` when `size` is reduced — leaving them would have made the pools 2/2, where any single OSD down blocks IO for its PGs. At size 3 the default `min_size` of 2 is correct. ## Left alone - `cephfs-ci` stays at size 2 — throwaway Woodpecker CI workspaces, at 2 rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop` refuse for every OSD. Unrelated to #434. - `objectstore` metadataPool is size 2 with no rationale recorded and an erasure-coded 2+1 dataPool. Flagging rather than changing, since I could not establish whether that 2 was deliberate. ## Expect on merge Re-replication of roughly 280 GiB — comparable to the backfill that just completed, so budget similar time and watch `ceph -s` rather than merging and walking away. The cluster is currently 217 PGs `active+clean` with all three OSDs up.

claude-bot added 1 commit 2026-08-26 00:13:48 +00:00

feat(rook-ceph): restore replicapool and cephfs-data0 to size 3

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

d376f318c2

These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.

The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.

The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.

cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.

cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot referenced this pull request

2026-08-26 00:13:58 +00:00

feat(rook-ceph): restore cephfs-metadata to size 3 #609

k.renner merged commit 02817ef45e into master

2026-08-26 00:14:06 +00:00

k.renner referenced this issue from a commit

2026-08-26 00:14:07 +00:00

Merge pull request 'feat(rook-ceph): restore replicapool and cephfs-data0 to size 3' (#610) from restore-replicapool-size into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#610