feat(rook-ceph): add an unreplicated CephFS for CI workspaces #542

2026-08-21T06:56:35Z

claude-bot commented

2026-08-21 06:56:35 +00:00

What

A separate CephFS (cephfs-ci) with both pools at size: 1, plus a ceph-filesystem-ci StorageClass. Nothing uses it yet — this commit is inert.

Why

Woodpecker creates a PVC per pipeline, fills it with a checkout plus build output, and deletes it when the pipeline ends. Those workspaces sit on the shared cephfs, so throwaway data is replicated as if it mattered — 2x in cephfs-data0, 3x in cephfs-metadata.

Measured

Polling both pools every 15 seconds. The interval is the point: at 5 minutes a whole pipeline is invisible, because its PVC is created, used and deleted inside ~2 minutes. Sampling that slowly reported ci_pods=0 during bursts CI was causing, which sent this investigation down three wrong paths before the cadence was fixed.

CI ACTIVE   1.5 min (6 intervals):  data0 143.2 MB   metadata 17.2 MB
CI IDLE    16.0 min (63 intervals): data0   0.0 MB   metadata 11.0 MB

CI share of writes:  data0 100%   metadata 61%

cephfs-data0 is 100% CI — zero bytes across 16 minutes of idle. Between pipelines the metadata pool is quiet at ~4 MB/day.

Per pipeline, on the shared filesystem: 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one: ~80 MB.

Why size 1 here, having been rejected for `cephfs`

cephfs keeps metadataPool at 3 because losing it destroys the whole filesystem rather than the objects on one OSD, and 550 GiB of real data depends on it. None of that applies here: every byte is a fresh git clone, and total loss means in-flight pipelines fail and get re-run. Replicating throwaway data was the cost, so it is what gets removed.

Losing an OSD takes these PGs offline and kills running pipelines — not a regression, since CI already breaks during Ceph maintenance.

What is deliberately NOT changed

ceph-filesystem-ephemeral stays pointing at the main filesystem. It has a second consumer: velero/restore-test.sh restores real data through it for verification, which must not land on unreplicated pools. A new class also sidesteps StorageClass.parameters being immutable.

Known consequence, deliberately not silenced here

size: 1 raises POOL_NO_REDUNDANCY, so cluster health will warn once this lands. mon_warn_on_pool_no_redundancy=false would silence it, but that is cluster-wide and would equally hide a production pool dropping to 1. That wants a targeted alert excluding the CI pools instead, so it is left out rather than switched off blind. Expect a HEALTH_WARN after merge.

Validation

kubectl kustomize builds clean (15 resources)
kubectl apply --dry-run=server: cephfilesystem/cephfs-ci created, storageclass/ceph-filesystem-ci created
requireSafeReplicaSize confirmed present in the live Rook CRD schema, not assumed
MDS sized 512Mi/1Gi with no standby — the 4Gi active+standby on cephfs is driven by immich cached inodes; a CI workspace is a few thousand files that vanish with the pipeline

## What A separate CephFS (`cephfs-ci`) with **both pools at `size: 1`**, plus a `ceph-filesystem-ci` StorageClass. Nothing uses it yet — this commit is inert. ## Why Woodpecker creates a PVC per pipeline, fills it with a checkout plus build output, and deletes it when the pipeline ends. Those workspaces sit on the shared `cephfs`, so **throwaway data is replicated as if it mattered** — 2x in `cephfs-data0`, 3x in `cephfs-metadata`. ## Measured Polling both pools every **15 seconds**. The interval is the point: at 5 minutes a whole pipeline is invisible, because its PVC is created, used and deleted inside ~2 minutes. Sampling that slowly reported `ci_pods=0` during bursts CI was causing, which sent this investigation down three wrong paths before the cadence was fixed. ``` CI ACTIVE 1.5 min (6 intervals): data0 143.2 MB metadata 17.2 MB CI IDLE 16.0 min (63 intervals): data0 0.0 MB metadata 11.0 MB CI share of writes: data0 100% metadata 61% ``` **`cephfs-data0` is 100% CI** — zero bytes across 16 minutes of idle. Between pipelines the metadata pool is quiet at ~4 MB/day. Per pipeline, on the shared filesystem: `71.6x2 + 8.2x3 = ~168 MB` of drive writes. On this one: **~80 MB**. ## Why size 1 here, having been rejected for `cephfs` `cephfs` keeps `metadataPool` at 3 because losing it destroys the **whole filesystem** rather than the objects on one OSD, and 550 GiB of real data depends on it. None of that applies here: every byte is a fresh git clone, and total loss means in-flight pipelines fail and get re-run. Replicating throwaway data was the cost, so it is what gets removed. Losing an OSD takes these PGs offline and kills running pipelines — not a regression, since CI already breaks during Ceph maintenance. ## What is deliberately NOT changed `ceph-filesystem-ephemeral` stays pointing at the main filesystem. It has a second consumer: `velero/restore-test.sh` restores **real data** through it for verification, which must not land on unreplicated pools. A new class also sidesteps `StorageClass.parameters` being immutable. ## Known consequence, deliberately not silenced here `size: 1` raises **`POOL_NO_REDUNDANCY`**, so cluster health will warn once this lands. `mon_warn_on_pool_no_redundancy=false` would silence it, but that is cluster-wide and would equally hide a production pool dropping to 1. That wants a targeted alert excluding the CI pools instead, so it is left out rather than switched off blind. **Expect a HEALTH_WARN after merge.** ## Validation - `kubectl kustomize` builds clean (15 resources) - `kubectl apply --dry-run=server`: `cephfilesystem/cephfs-ci created`, `storageclass/ceph-filesystem-ci created` - `requireSafeReplicaSize` confirmed present in the live Rook CRD schema, not assumed - MDS sized 512Mi/1Gi with no standby — the 4Gi active+standby on `cephfs` is driven by immich cached inodes; a CI workspace is a few thousand files that vanish with the pipeline ## Next Repointing the Woodpecker agent to `ceph-filesystem-ci` is a separate change, once this is confirmed healthy. Staged deliberately so the filesystem lands empty and unused first.

claude-bot added 1 commit 2026-08-21 06:56:35 +00:00

feat(rook-ceph): add an unreplicated CephFS for CI workspaces

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

503d2a4876

Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.

Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.

  one pipeline, a single 15s interval:  data0 +71.6 MB   metadata +8.2 MB
  every interval with no CI pod:        data0   ~0        metadata   ~0

So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.

size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.

`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.

Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.

Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.

The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.

review-bot reviewed

2026-08-21 07:00:11 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 503d2a48762f0c34490e50e1b80a20890de43239.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a new, currently-unused CephFS (cephfs-ci, both pools size: 1) plus a ceph-filesystem-ci StorageClass for future Woodpecker CI workspaces, wiring it into kustomization.yaml. The change is purely additive (86 new lines in a new file, +1 in kustomization.yaml), nothing references the new class yet, and the reasoning in the manifest comments and PR body is well grounded in measured data. I found no blocking issues.

Verified against the checkout:

ceph-filesystem-ephemeral (storageclass.yaml:154) is untouched and still targets the replicated cephfs/cephfs-data0, and velero/restore-test.sh:238 does indeed use ceph-filesystem-ephemeral for real-data restore verification — the PR's claim about why that class can't be repointed checks out.
cephfs-ci's data pool naming (data0 → resulting pool cephfs-ci-data0, referenced as pool: cephfs-ci-data0 in the StorageClass) mirrors the exact convention already used by cephfs/cephfs-data0 in storageclass.yaml:99-136.
CephHealthWarn in monitoring-rules/prometheusrule-ceph.yaml:14-21 fires on ceph_health_status == 1 for any condition that doesn't self-clear within 6h — confirming the PR's own "expect a HEALTH_WARN after merge" is accurate, and that this won't be silently swallowed.
No naming collisions: cephfs-ci / ceph-filesystem-ci don't appear anywhere else in the tree before this PR.

Tests

This is an infrastructure-as-code repo; the relevant gate is .woodpecker/ci-manifests-test.yml's kustomize-build step, which runs kubectl kustomize over every kustomization directory including this one — it will catch structural breakage (bad resource reference, malformed YAML) but, being client-side, cannot validate CRD field names like requireSafeReplicaSize against the live schema. The PR body states that was checked separately against the live Rook CRD and via kubectl apply --dry-run=server; I can't re-run that from this checkout, but the claim is specific and plausible, and the exact same replicated:/requireSafeReplicaSize shape is not used elsewhere in the repo to cross-check against. No functional consumer exists yet, so there's nothing to add integration test coverage for — appropriate for a deliberately inert, staged change.

🟡 Suggestions

cephfs-ci.yaml:50-56 — the MDS spec omits priorityClassName, unlike cephfs's MDS (storageclass.yaml:110-113), which explicitly sets system-node-critical even though cephcluster.yaml:164-165 already sets priorityClassNames.all: system-node-critical for the cluster. That the main filesystem bothers to set it explicitly suggests priorityClassNames.all may not cascade to CephFilesystem/MDS pods in this Rook version — if that's right, cephfs-ci's MDS correctly stays evictable/non-critical, matching the PR's intent. If all does cascade to MDS pods, the CI MDS would inherit system-node-critical and undermine the "this is fine to lose" design. Worth a one-time check after merge: kubectl -n rook-ceph get pod -l app=rook-ceph-mds,rook_file_system=cephfs-ci -o jsonpath='{.items[0].spec.priorityClassName}'.
The POOL_NO_REDUNDANCY warning is explicitly deferred rather than fixed, which is reasonable, but given CephHealthWarn's 6h non-self-clearing threshold, this will become a permanent firing alert once cephfs-ci exists (not a one-time notification) until the targeted alert-exclusion mentioned in the PR body is actually built. Worth turning that "Next" item into a tracked follow-up so the warning doesn't just get muscle-memory-dismissed indefinitely.

🟢 Nits

None.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `503d2a48762f0c34490e50e1b80a20890de43239`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a new, currently-unused CephFS (`cephfs-ci`, both pools `size: 1`) plus a `ceph-filesystem-ci` StorageClass for future Woodpecker CI workspaces, wiring it into `kustomization.yaml`. The change is purely additive (86 new lines in a new file, +1 in `kustomization.yaml`), nothing references the new class yet, and the reasoning in the manifest comments and PR body is well grounded in measured data. I found no blocking issues. Verified against the checkout: - `ceph-filesystem-ephemeral` (`storageclass.yaml:154`) is untouched and still targets the replicated `cephfs`/`cephfs-data0`, and `velero/restore-test.sh:238` does indeed use `ceph-filesystem-ephemeral` for real-data restore verification — the PR's claim about why that class can't be repointed checks out. - `cephfs-ci`'s data pool naming (`data0` → resulting pool `cephfs-ci-data0`, referenced as `pool: cephfs-ci-data0` in the StorageClass) mirrors the exact convention already used by `cephfs`/`cephfs-data0` in `storageclass.yaml:99-136`. - `CephHealthWarn` in `monitoring-rules/prometheusrule-ceph.yaml:14-21` fires on `ceph_health_status == 1` for any condition that doesn't self-clear within 6h — confirming the PR's own "expect a HEALTH_WARN after merge" is accurate, and that this won't be silently swallowed. - No naming collisions: `cephfs-ci` / `ceph-filesystem-ci` don't appear anywhere else in the tree before this PR. ## Tests This is an infrastructure-as-code repo; the relevant gate is `.woodpecker/ci-manifests-test.yml`'s `kustomize-build` step, which runs `kubectl kustomize` over every kustomization directory including this one — it will catch structural breakage (bad resource reference, malformed YAML) but, being client-side, cannot validate CRD field names like `requireSafeReplicaSize` against the live schema. The PR body states that was checked separately against the live Rook CRD and via `kubectl apply --dry-run=server`; I can't re-run that from this checkout, but the claim is specific and plausible, and the exact same `replicated:`/`requireSafeReplicaSize` shape is not used elsewhere in the repo to cross-check against. No functional consumer exists yet, so there's nothing to add integration test coverage for — appropriate for a deliberately inert, staged change. ## 🟡 Suggestions - **`cephfs-ci.yaml:50-56`** — the MDS spec omits `priorityClassName`, unlike `cephfs`'s MDS (`storageclass.yaml:110-113`), which explicitly sets `system-node-critical` even though `cephcluster.yaml:164-165` already sets `priorityClassNames.all: system-node-critical` for the cluster. That the main filesystem bothers to set it explicitly suggests `priorityClassNames.all` may *not* cascade to CephFilesystem/MDS pods in this Rook version — if that's right, `cephfs-ci`'s MDS correctly stays evictable/non-critical, matching the PR's intent. If `all` *does* cascade to MDS pods, the CI MDS would inherit `system-node-critical` and undermine the "this is fine to lose" design. Worth a one-time check after merge: `kubectl -n rook-ceph get pod -l app=rook-ceph-mds,rook_file_system=cephfs-ci -o jsonpath='{.items[0].spec.priorityClassName}'`. - The `POOL_NO_REDUNDANCY` warning is explicitly deferred rather than fixed, which is reasonable, but given `CephHealthWarn`'s 6h non-self-clearing threshold, this will become a **permanent** firing alert once `cephfs-ci` exists (not a one-time notification) until the targeted alert-exclusion mentioned in the PR body is actually built. Worth turning that "Next" item into a tracked follow-up so the warning doesn't just get muscle-memory-dismissed indefinitely. ## 🟢 Nits None. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit baa5e62bbe into master

2026-08-21 07:27:48 +00:00

k.renner referenced this issue from a commit

2026-08-21 07:27:48 +00:00

Merge pull request 'feat(rook-ceph): add an unreplicated CephFS for CI workspaces' (#542) from ci-throwaway-cephfs into master

claude-bot referenced this issue from a commit

2026-08-21 07:31:12 +00:00

perf(woodpecker): put CI workspaces on the unreplicated CephFS

claude-bot referenced this pull request

2026-08-21 07:31:26 +00:00

perf(woodpecker): put CI workspaces on the unreplicated CephFS #543

claude-bot referenced this issue from a commit