perf(woodpecker): put CI workspaces on the unreplicated CephFS #543

2026-08-21T07:31:26Z

claude-bot commented

2026-08-21 07:31:26 +00:00

What

Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci — the dedicated size-1 filesystem from #542.

Verified before repointing

#542 landed inert by design, so the filesystem was confirmed healthy first:

cephfs-ci            PHASE Ready, ACTIVEMDS 1
rook-ceph-mds-cephfs-ci-a   Running
cephfs-ci-metadata   size: 1
cephfs-ci-data0      size: 1

Measured

Polling both CephFS pools every 15 seconds:

CI ACTIVE   1.5 min (6 intervals):  data0 143.2 MB   metadata 17.2 MB
CI IDLE    16.0 min (63 intervals): data0   0.0 MB   metadata 11.0 MB

CI share of writes:  data0 100%   metadata 61%

cephfs-data0 takes zero bytes across 16 minutes without a pipeline. The 15s cadence is what made this visible at all — a pipeline creates, uses and deletes its PVC inside ~2 minutes, so 5-minute sampling reported ci_pods=0 during bursts CI was causing.

Per pipeline: ~168 MB → ~80 MB of drive writes.

Unchanged

STORAGE_RWX=true stays. Pipeline steps share the workspace PVC and the k8s backend can schedule them on different nodes, so RWO fails to mount on the second node. Only the filesystem underneath changes.

Correction to #542

That PR justified leaving ceph-filesystem-ephemeral alone by claiming restore-test.sh handles "REAL data which must not land on unreplicated pools". That reasoning is wrong, and this PR fixes the comment.

Those probe PVCs are exactly as throwaway as a CI workspace — the authoritative copy is the backup on Hetzner, and losing a probe just means re-running the test. The script own comment already calls them throwaway.

The real reason it cannot move is technical: the probes are snapshot clones, and ceph fs subvolume snapshot clone operates within a single filesystem, so a snapshot taken in cephfs cannot be cloned into cephfs-ci. Landing them unreplicated would need a second size-1 data pool inside cephfs — and restore-test runs weekly against CI ~96 pipelines/day, so that is not worth the complexity.

What to watch after merge

First pipeline should provision its PVC from ceph-filesystem-ci and land in cephfs-ci-data0
cephfs-data0 write volume should drop to near zero
HEALTH_WARN ... 2 pool(s) have no replicas configured is expected and already present from #542

## What Repoints `WOODPECKER_BACKEND_K8S_STORAGE_CLASS` at `ceph-filesystem-ci` — the dedicated size-1 filesystem from #542. ## Verified before repointing #542 landed inert by design, so the filesystem was confirmed healthy first: ``` cephfs-ci PHASE Ready, ACTIVEMDS 1 rook-ceph-mds-cephfs-ci-a Running cephfs-ci-metadata size: 1 cephfs-ci-data0 size: 1 ``` ## Measured Polling both CephFS pools every **15 seconds**: ``` CI ACTIVE 1.5 min (6 intervals): data0 143.2 MB metadata 17.2 MB CI IDLE 16.0 min (63 intervals): data0 0.0 MB metadata 11.0 MB CI share of writes: data0 100% metadata 61% ``` `cephfs-data0` takes **zero bytes** across 16 minutes without a pipeline. The 15s cadence is what made this visible at all — a pipeline creates, uses and deletes its PVC inside ~2 minutes, so 5-minute sampling reported `ci_pods=0` during bursts CI was causing. Per pipeline: **~168 MB → ~80 MB** of drive writes. ## Unchanged `STORAGE_RWX=true` stays. Pipeline steps share the workspace PVC and the k8s backend can schedule them on different nodes, so RWO fails to mount on the second node. Only the filesystem underneath changes. ## Correction to #542 That PR justified leaving `ceph-filesystem-ephemeral` alone by claiming `restore-test.sh` handles *"REAL data which must not land on unreplicated pools"*. **That reasoning is wrong**, and this PR fixes the comment. Those probe PVCs are exactly as throwaway as a CI workspace — the authoritative copy is the backup on Hetzner, and losing a probe just means re-running the test. The script own comment already calls them throwaway. The real reason it cannot move is technical: the probes are snapshot **clones**, and `ceph fs subvolume snapshot clone` operates within a single filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`. Landing them unreplicated would need a second size-1 data pool inside `cephfs` — and restore-test runs **weekly** against CI ~96 pipelines/day, so that is not worth the complexity. ## What to watch after merge - First pipeline should provision its PVC from `ceph-filesystem-ci` and land in `cephfs-ci-data0` - `cephfs-data0` write volume should drop to near zero - `HEALTH_WARN ... 2 pool(s) have no replicas configured` is expected and already present from #542

claude-bot added 1 commit 2026-08-21 07:31:27 +00:00

perf(woodpecker): put CI workspaces on the unreplicated CephFS

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

7666d220c5

Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci, the
dedicated filesystem added in #542 whose pools are both size 1.

Measured 2026-08-21, polling both CephFS pools every 15s:

  CI active   1.5 min (6 intervals):  data0 143.2 MB   metadata 17.2 MB
  CI idle    16.0 min (63 intervals): data0   0.0 MB   metadata 11.0 MB
  CI share of writes: data0 100%, metadata 61%

data0 takes zero bytes across 16 minutes without a pipeline, so its entire
write volume is CI. The 15s cadence is what made this visible: a pipeline
creates, uses and deletes its PVC inside ~2 minutes, so a 5-minute sample
reported no CI pods during bursts CI was causing.

Per pipeline this moves the drive cost from 71.6x2 + 8.2x3 = ~168 MB to
~80 MB, since neither pool is replicated on the new filesystem.

RWX is unchanged and still required: pipeline steps share the workspace PVC
and the k8s backend can schedule them on different nodes, so an RWO volume
fails to mount on the second node. Only the filesystem underneath changes.

Also corrects the justification written into #542 for leaving
ceph-filesystem-ephemeral alone. That commit claimed restore-test.sh handles
"REAL data which must not land on unreplicated pools". That is wrong: its
probe PVCs are as throwaway as a CI workspace -- the authoritative copy is the
backup on Hetzner, and losing a probe just means re-running the test.

The actual reason it cannot move is technical: those probes are snapshot
CLONES, and `ceph fs subvolume snapshot clone` operates within a single
filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
Landing them unreplicated would require a second size-1 data pool inside
`cephfs`; restore-test runs weekly against CI's ~96 pipelines/day, so that is
not worth the complexity.

cephfs-ci verified healthy before this change: PHASE Ready, MDS Running, both
pools reporting size 1.

k.renner merged commit a11493ee22 into master

2026-08-21 07:32:11 +00:00

k.renner referenced this issue from a commit

2026-08-21 07:32:13 +00:00

Merge pull request 'perf(woodpecker): put CI workspaces on the unreplicated CephFS' (#543) from repoint-ci-workspaces into master

review-bot reviewed

2026-08-21 07:33:54 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 7666d220c55442e4c9f46d97513efeb37a7ac780.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR repoints Woodpecker CI workspaces to a dedicated size-1 CephFS filesystem, reducing per-pipeline write volume from ~168 MB to ~80 MB by avoiding unnecessary replication of throwaway data. The infrastructure was verified healthy before this change, and the accompanying comment clarifies why the restore-test filesystem cannot be repointed (snapshot clones require same-filesystem operations, not data-safety concerns).

🔴 Blocking

None. No blocking issues found.

🟡 Suggestions

None. The change is well-executed.

🟢 Nits

None.

Tests

This is a configuration change to a deployed system, not testable via unit tests. The PR describes sufficient post-merge verification: the first pipeline should provision from ceph-filesystem-ci, cephfs-data0 write volume should drop to near zero, and a HEALTH_WARN for two unreplicated pools is expected (already present from #542).

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `7666d220c55442e4c9f46d97513efeb37a7ac780`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR repoints Woodpecker CI workspaces to a dedicated size-1 CephFS filesystem, reducing per-pipeline write volume from ~168 MB to ~80 MB by avoiding unnecessary replication of throwaway data. The infrastructure was verified healthy before this change, and the accompanying comment clarifies why the restore-test filesystem cannot be repointed (snapshot clones require same-filesystem operations, not data-safety concerns). ## 🔴 Blocking None. No blocking issues found. ## 🟡 Suggestions None. The change is well-executed. ## 🟢 Nits None. ## Tests This is a configuration change to a deployed system, not testable via unit tests. The PR describes sufficient post-merge verification: the first pipeline should provision from `ceph-filesystem-ci`, `cephfs-data0` write volume should drop to near zero, and a `HEALTH_WARN` for two unreplicated pools is expected (already present from #542). ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#543