konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

feat(rook-ceph): add EC 2+1 bulk data pool to cephfs #953

Merged
k.renner merged 1 commits from feat/cephfs-ec-bulk into master 2026-09-25 05:38:39 +00:00
Conversation 1 Commits 1 Files Changed 2
+40
claude-bot commented 2026-09-25 05:29:31 +00:00
Member
Copy Link
Copy Source

Adds an in-cluster slow storage tier. The alternative, reviving nas-1, was turned down because of its power draw.

Changes

  • CephFilesystem cephfs: secondary data pool bulk (EC 2+1, failureDomain host). data0 stays the default data pool, which Rook requires for EC on CephFS.
  • New StorageClass ceph-filesystem-bulk (pool cephfs-bulk, Retain).
  • New empty PVC audiobookshelf-books-bulk (300Gi, RWX). Nothing mounts it yet.

Why

Ceph is 62% raw-full (1.79 of 2.88 TB). Bulk media at 1.5x instead of 3x overhead. Audiobookshelf goes first (81 GiB used, about 120 GB raw saved); the Immich library and Paperless media follow once this is proven.

Behaviour during reboots

Same as the existing migration.rgw.buckets.data EC 2+1 pool (checked live: ec_overwrites set, min_size 2). IO continues through a single-node reboot, with no redundancy for that window. Ceph docs recommend min_size k+1; that would block IO on every rolling reboot with only 3 hosts.

Verification

  • kubectl apply --dry-run=server passes for storageclass.yaml.
  • After merge: ceph fs ls lists cephfs-bulk and the PVC is Bound.

Follow-up (separate PR)

Copy the books with ABS scaled to 0 and the libation CronJob suspended, then point the Deployment and CronJob at audiobookshelf-books-bulk.

🤖 Generated with Claude Code

Adds an in-cluster slow storage tier. The alternative, reviving nas-1, was turned down because of its power draw. ## Changes - `CephFilesystem cephfs`: secondary data pool `bulk` (EC 2+1, failureDomain host). `data0` stays the default data pool, which Rook requires for EC on CephFS. - New StorageClass `ceph-filesystem-bulk` (pool `cephfs-bulk`, Retain). - New empty PVC `audiobookshelf-books-bulk` (300Gi, RWX). Nothing mounts it yet. ## Why Ceph is 62% raw-full (1.79 of 2.88 TB). Bulk media at 1.5x instead of 3x overhead. Audiobookshelf goes first (81 GiB used, about 120 GB raw saved); the Immich library and Paperless media follow once this is proven. ## Behaviour during reboots Same as the existing `migration.rgw.buckets.data` EC 2+1 pool (checked live: `ec_overwrites` set, `min_size 2`). IO continues through a single-node reboot, with no redundancy for that window. Ceph docs recommend min_size k+1; that would block IO on every rolling reboot with only 3 hosts. ## Verification - `kubectl apply --dry-run=server` passes for storageclass.yaml. - After merge: `ceph fs ls` lists `cephfs-bulk` and the PVC is Bound. ## Follow-up (separate PR) Copy the books with ABS scaled to 0 and the libation CronJob suspended, then point the Deployment and CronJob at `audiobookshelf-books-bulk`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 1 commit 2026-09-25 05:29:31 +00:00
feat(rook-ceph): add EC 2+1 bulk data pool to cephfs
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
ci/woodpecker/pr/pr-review Pipeline was successful
Details
52b294ed22 
Adds a secondary erasure-coded (2+1, failureDomain host) data pool
'bulk' to the cephfs filesystem and a ceph-filesystem-bulk StorageClass
targeting it. data0 stays the default data pool, as Rook requires for
EC CephFS pools. Bulk media at 1.5x instead of 3x raw overhead.

Behaviour matches the existing migration.rgw.buckets.data EC pool:
Rook sets ec_overwrites, min_size 2 (=k), so IO continues through a
single-node reboot without redundancy for that window.

Also provisions an empty audiobookshelf-books-bulk PVC on the new
class. Deployment and libation CronJob still mount audiobookshelf-books;
the data copy and cutover follow separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 52b294ed22240ad0ad85ef7c0311da5c46152962.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a second, erasure-coded (2+1) CephFS data pool (bulk) alongside the existing replicated data0 pool, a matching ceph-filesystem-bulk StorageClass, and a new empty, unmounted 300Gi PVC for Audiobookshelf. No workload is wired to the new PVC yet, so this PR carries no runtime blast radius on its own. No blocking issues found.

🟡 Suggestions

  • kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:157 — The PR body justifies the reboot-tolerance of the new bulk pool by analogy to the existing migration.rgw.buckets.data EC 2+1 pool ("checked live: ec_overwrites set, min_size 2"), but neither pool sets these via the CR (no parameters: block in either the new dataPools[].bulk here or dataPool in objectstore.yaml), so both rely on Rook/Ceph defaults for a freshly created EC 2+1 pool. That's a reasonable inference, but it's still carried over from one pool type (RGW CephObjectStore) to another (CephFilesystem data pool) — worth confirming directly on cephfs-bulk after merge (ceph osd pool get cephfs-bulk allow_ec_overwrites and min_size) rather than assuming parity, since the two are created through different Rook reconcile paths. The PR's stated post-merge verification (ceph fs ls, PVC Bound) wouldn't catch a wrong min_size/overwrite setting — only an explicit ceph osd pool get or a live-write-during-reboot test would.

Tests

No automated tests apply to this repo's Kubernetes manifests beyond the kubectl apply --dry-run=server check the author already ran. The change is inert until merged (nothing consumes the new PVC/StorageClass yet), and the follow-up PR that points a workload at audiobookshelf-books-bulk is explicitly deferred — that's the right place to validate real read/write behavior on the EC pool. Recommend the ceph osd pool get checks above happen as part of this PR's verification step (or the follow-up) rather than being skipped.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `52b294ed22240ad0ad85ef7c0311da5c46152962`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a second, erasure-coded (2+1) CephFS data pool (`bulk`) alongside the existing replicated `data0` pool, a matching `ceph-filesystem-bulk` StorageClass, and a new empty, unmounted 300Gi PVC for Audiobookshelf. No workload is wired to the new PVC yet, so this PR carries no runtime blast radius on its own. No blocking issues found. ## 🟡 Suggestions - `kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:157` — The PR body justifies the reboot-tolerance of the new `bulk` pool by analogy to the existing `migration.rgw.buckets.data` EC 2+1 pool ("checked live: `ec_overwrites` set, `min_size 2`"), but neither pool sets these via the CR (no `parameters:` block in either the new `dataPools[].bulk` here or `dataPool` in `objectstore.yaml`), so both rely on Rook/Ceph defaults for a freshly created EC 2+1 pool. That's a reasonable inference, but it's still carried over from one pool type (RGW `CephObjectStore`) to another (`CephFilesystem` data pool) — worth confirming directly on `cephfs-bulk` after merge (`ceph osd pool get cephfs-bulk allow_ec_overwrites` and `min_size`) rather than assuming parity, since the two are created through different Rook reconcile paths. The PR's stated post-merge verification (`ceph fs ls`, PVC `Bound`) wouldn't catch a wrong `min_size`/overwrite setting — only an explicit `ceph osd pool get` or a live-write-during-reboot test would. ## Tests No automated tests apply to this repo's Kubernetes manifests beyond the `kubectl apply --dry-run=server` check the author already ran. The change is inert until merged (nothing consumes the new PVC/StorageClass yet), and the follow-up PR that points a workload at `audiobookshelf-books-bulk` is explicitly deferred — that's the right place to validate real read/write behavior on the EC pool. Recommend the `ceph osd pool get` checks above happen as part of this PR's verification step (or the follow-up) rather than being skipped. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=52b294ed22240ad0ad85ef7c0311da5c46152962 -->
k.renner merged commit 26e3dad557 into master 2026-09-25 05:38:39 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#953