feat(paperless): provision RBD volume to move data off CephFS (phase 1) #533
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#533
Reference in New Issue
Block a user
Refs #434. Chasing the
cephfs-metadatawrite baseline.Agreed: metadata stays at size 3
Dropping
cephfs-metadatato size 2 would save ~7 GB/day, and it is the wrong trade — losing that pool destroys the entire filesystem, not just the objects on one OSD, and the 275 GiB incephfs-data0rides on it. Cutting writes at source instead.The finding
paperless-dataisReadWriteManyonceph-filesystem, but only one pod mounts it:Contents are all single-writer state — 50 MB Whoosh index,
classification_model.pickle,celerybeat-schedule.db,log/,migration_lock. 75 files, nothing shared. It is on CephFS by default, not by requirement.What it costs
Measured with zero CI pipelines running:
Two directory commits against 2,718 journal events means this is cap/inode churn from repeated access, not file writes. It accounts for most of the constant 7-8 GB/day on
cephfs-metadata→ ~21-24 GB/day on the drives at size 3.Phase 1 is inert
The PVC is created empty and mounted by nothing. Merging it changes no behaviour.
Phase 2 (copy) and phase 3 (repoint) are separate and documented in
MIGRATION-data-to-block.md, including why the copy needs paperless stopped: the Whoosh index andcelerybeat-schedule.dbare both live-written, so copying under an active writer gives a torn read.The old PVC is
prune: disabledand stays as the rollback until search is confirmed working — that is the check that proves the index survived.Honest sizing
~21-24 GB/day off the drives against ~432 GB/day cluster-wide — about 5%. This is a correctness fix (a single-writer volume does not belong on a shared filesystem) more than a large win.
It also removes 22 req/sec of MDS load, which benefits the workloads that genuinely need CephFS — Woodpecker CI workspaces share the same MDS.
Caveat on my own track record here
I have been wrong three times in this thread about the metadata source: first "idle filesystem", then "CI workspaces", then "consumer polling". Each was disproved by the next measurement. This one is better grounded — single mounter verified, contents enumerated, request rate matching the MDS delta — but I will confirm the metadata rate actually drops after phase 3 rather than assume it.
🤖 Generated with Claude Code
paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod mounts it -- the paperless Deployment itself. Its contents are all single-writer state: the 50 MB Whoosh index, classification_model.pickle, celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only `consume` and `media` genuinely need RWX, because the ftp-scanner writes to those from another node. It is on CephFS by default, not by requirement, and every access goes through the MDS. Measured 2026-08-20 with ZERO CI pipelines running: paperless-data client request_load_avg 22 (all other sessions 0-1) MDS over 180s 4,052 client requests = 22.5/s 2,718 journal events = 15/s 2 dir commits <- not file writes Cap and inode churn from repeated access, not data being written. That accounts for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is size 3, roughly 21-24 GB/day reaches the drives. cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0 rides on it. Cutting writes at source is the fix; cutting replication is not. PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data copy and the deployment repoint are separate steps, documented in MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the Whoosh index and celerybeat schedule are both live-written, so copying under an active writer gives a torn read). Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so about 5%. This is a correctness fix -- a single-writer volume does not belong on a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS load, which benefits Woodpecker CI workspaces sharing the same MDS. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commitd2d06d853234428f4990a074ed6aded50ca42592.Summary
This PR provisions the first phase of a multi-phase migration to move
paperless-datafrom CephFS (ReadWriteMany) to RBD (ReadWriteOnce) block storage. Phase 1 adds an empty PVC that no workload mounts; merging it changes no behavior. The rationale is well-documented and grounded in measurements: a single-reader volume generating 22 req/sec of MDS load and ~7-8 GB/day of metadata churn belongs on block storage, not a shared filesystem.🟢 No Blocking Issues
The changes are safe to merge:
paperless-data-blockPVC is created empty and unmounted (inert)ceph-blockstorage class is defined inkubernetes/infrastructure/rook-ceph-cluster/storageclass.yamlprune: disabledannotation correctly prevents Flux from deleting the PVC during intermediate phasespaperless-dataPVC is retained as a rollback mechanism (itself hasprune: disabled)🟢 Design & Documentation
Strengths:
MIGRATION-data-to-block.mdcelerybeat-schedule.dbare live-written), addressed by scaling paperless to 0pvcs.yamlexplain why this volume doesn't belong on a shared filesystem🟡 Suggestions
Minor observations for future phases (not blockers for Phase 1):
Phase 2 execution: The migration guide depends on paperless scaling cleanly to 0 and the one-off copy pod finding a node with
ceph-blockattachment capability. Worth a quick dry-run of the copy pod definition once Phase 2 is drafted.Verification sequence (Phase 3): The guide lists three checks (search results, MDS session gone, metadata write rate drop). The search test is the highest-confidence proof of index integrity; consider running it first to short-circuit if corruption is detected.
Documentation link: The inline comment in
pvcs.yamlreferencesMIGRATION-data-to-block.md— once Phase 2 PR lands, it may be helpful to cross-link (or symlink) from the deployment to both phases so the full sequence is visible in one place.Tests
Phase 1 adds no code or logic; the tests that matter are manual verification once deployed:
kubectl -n paperless get pvc paperless-data-block→ BoundRecommendation
Looks reasonable — no blocking issues. Phase 1 is inert; proceed with confidence. Phases 2–3 are well-scoped and documented for a future PR.