KubePersistentVolumeFillingUp is firing for paperclip/paperclip-data. The 20Gi RWO volume is mounted at /paperclip and shared by every agent that runs inside the paperclip container. Only about 2.19 GB is free, and space has been dropping by several GB per day. This PR grows the PVC to 40Gi. The ceph-block StorageClass supports online expansion, so the volume should not run out before someone can look at it. This buys time. It does not fix the root cause. We still don't know what is filling the volume, and the diag account can't see inside it (no exec). The manual steps below are still needed.
Changes
kubernetes/apps/paperclip/pvc.yaml: resources.requests.storage goes from 20Gi to 40Gi, with a short comment that points to this issue.
Verification
Checked live this run:
kubectl -n paperclip get pvc,pods -o wide: paperclip-data is Bound with 20Gi on ceph-block and is 12 days old. paperclip-b65cc7768-msnkf is 1/1 Running with 0 restarts, on k8s-1.
Prometheus kubelet_volume_stats_available_bytes{persistentvolumeclaim="paperclip-data"} currently returns 2188918784 (about 2.19 GB). Over the last 12h (hourly samples) it fell from about 4.76 GB to 2.19 GB, with a drop of about 0.8 GB between 14:00Z and 15:00Z. The issue's numbers are correct.
kubectl get storageclass ceph-block -o jsonpath={.allowVolumeExpansion} returns true.
CephCluster status.ceph.capacity shows about 1.25 TB available out of about 2.88 TB raw. Adding 20Gi (about 60Gi raw at 3× replication) is easily covered.
After merge and Flux reconcile, check:
kubectl -n paperclip get pvc paperclip-data should show CAPACITY 40Gi. Look at kubectl -n paperclip describe pvc paperclip-data for FileSystemResizeSuccessful. ceph-csi resizes the filesystem online, so no pod restart should be needed.
kubelet_volume_stats_available_bytes should rise by about 20 GB, and the alert should clear once the linear prediction stops pointing at zero within 4 days. If the current write rate continues (about 2.5–5 GB/day), the alert will come back in about a week. That is why the cleanup below matters.
Required manual follow-up (root cause)
Now, before Flux applies this (about 10h of headroom at the last-24h rate), find out what is growing: kubectl -n paperclip exec deploy/paperclip -- du -xh -d2 /paperclip | sort -h | tail -20
Delete stale agent workspaces, repo checkouts, caches and run logs under /paperclip. Don't touch anything a run that is still active is using.
Add retention for agent workspaces/run artifacts. This belongs in the paperclip app or its config, which is outside this repo's scope. Without it, the bigger volume only delays the next alert.
Notes
The issue's facts held up when I checked them live. Its suggested repo fix (expand to 40Gi) is the change made here. The cause (workspace/cache buildup) is still unconfirmed: I couldn't exec into the pod to check it.
The PVC has kustomize.toolkit.fluxcd.io/prune: disabled. That only blocks deletion, so Flux still applies the spec change. Kubernetes PVCs can only grow, so going back to 20Gi later is not possible.
Separate finding: kubectl -n paperclip logs deploy/paperclip --tail=60 shows a tight retry loop, a few times a minute: sandbox callback bridge failed to write 504 backstop for e8395160-... (attempt n/3): ... sandboxes.agents.x-k8s.io "pc-01m4ak2swg1mxg6tzf1vy4454g" not found. This goes to container stdout, not to the PVC, so it is probably not why the volume is filling. It does look like an app bug: the bridge keeps retrying against a Sandbox that has already been deleted. The paperclip owner should look at it. The same log lines also show an API deprecation warning: agents.x-k8s.io/v1alpha1 Sandbox is deprecated; use v1beta1.
## What this PR does
`KubePersistentVolumeFillingUp` is firing for `paperclip/paperclip-data`. The 20Gi RWO volume is mounted at `/paperclip` and shared by every agent that runs inside the paperclip container. Only about 2.19 GB is free, and space has been dropping by several GB per day. This PR grows the PVC to 40Gi. The `ceph-block` StorageClass supports online expansion, so the volume should not run out before someone can look at it. **This buys time. It does not fix the root cause.** We still don't know what is filling the volume, and the diag account can't see inside it (no exec). The manual steps below are still needed.
## Changes
- `kubernetes/apps/paperclip/pvc.yaml`: `resources.requests.storage` goes from `20Gi` to `40Gi`, with a short comment that points to this issue.
## Verification
Checked live this run:
- `kubectl -n paperclip get pvc,pods -o wide`: `paperclip-data` is Bound with 20Gi on `ceph-block` and is 12 days old. `paperclip-b65cc7768-msnkf` is 1/1 Running with 0 restarts, on `k8s-1`.
- Prometheus `kubelet_volume_stats_available_bytes{persistentvolumeclaim="paperclip-data"}` currently returns `2188918784` (about 2.19 GB). Over the last 12h (hourly samples) it fell from about 4.76 GB to 2.19 GB, with a drop of about 0.8 GB between 14:00Z and 15:00Z. The issue's numbers are correct.
- `kubectl get storageclass ceph-block -o jsonpath={.allowVolumeExpansion}` returns `true`.
- CephCluster `status.ceph.capacity` shows about 1.25 TB available out of about 2.88 TB raw. Adding 20Gi (about 60Gi raw at 3× replication) is easily covered.
After merge and Flux reconcile, check:
- `kubectl -n paperclip get pvc paperclip-data` should show CAPACITY `40Gi`. Look at `kubectl -n paperclip describe pvc paperclip-data` for `FileSystemResizeSuccessful`. ceph-csi resizes the filesystem online, so no pod restart should be needed.
- `kubelet_volume_stats_available_bytes` should rise by about 20 GB, and the alert should clear once the linear prediction stops pointing at zero within 4 days. If the current write rate continues (about 2.5–5 GB/day), the alert will come back in about a week. That is why the cleanup below matters.
## Required manual follow-up (root cause)
1. **Now, before Flux applies this** (about 10h of headroom at the last-24h rate), find out what is growing:
`kubectl -n paperclip exec deploy/paperclip -- du -xh -d2 /paperclip | sort -h | tail -20`
2. Delete stale agent workspaces, repo checkouts, caches and run logs under `/paperclip`. Don't touch anything a run that is still active is using.
3. Add retention for agent workspaces/run artifacts. This belongs in the paperclip app or its config, which is outside this repo's scope. Without it, the bigger volume only delays the next alert.
## Notes
- The issue's facts held up when I checked them live. Its suggested repo fix (expand to 40Gi) is the change made here. The cause (workspace/cache buildup) is still **unconfirmed**: I couldn't exec into the pod to check it.
- The PVC has `kustomize.toolkit.fluxcd.io/prune: disabled`. That only blocks deletion, so Flux still applies the spec change. Kubernetes PVCs can only grow, so going back to 20Gi later is not possible.
- Separate finding: `kubectl -n paperclip logs deploy/paperclip --tail=60` shows a tight retry loop, a few times a minute: `sandbox callback bridge failed to write 504 backstop for e8395160-... (attempt n/3): ... sandboxes.agents.x-k8s.io "pc-01m4ak2swg1mxg6tzf1vy4454g" not found`. This goes to container stdout, not to the PVC, so it is probably not why the volume is filling. It does look like an app bug: the bridge keeps retrying against a Sandbox that has already been deleted. The paperclip owner should look at it. The same log lines also show an API deprecation warning: `agents.x-k8s.io/v1alpha1 Sandbox is deprecated; use v1beta1`.
- No thresholds or alert rules were changed.
Closes #1184
Checked: diff is one line, pvc.yaml 20Gi→40Gi. Premise: alert fires on a real drain (Iris REN-1134: 3.18 GB free at 13:25Z; PR reads 2.19 GB at 15:00Z); PR claims headroom only, not root cause — correct scope. StorageClass ceph-block (repo storageclass.yaml): allowVolumeExpansion true, controller-expand secret set, ext4 → ceph-csi online resize, no pod restart. Precedent: gitea data-gitea-0 grown 40→60Gi via Flux (REN-583), live 60Gi now. Live: paperclip-data Bound 20Gi, prune: disabled (does not block spec apply). Flux paperclip Ready at 60e4ba4d. CI green (manifests, scripts, pr-review).
What breaks if wrong: resize fails → PVC sits at FileSystemResizePending, volume stays 20Gi and writable; no outage. Who notices: KubePersistentVolumeFillingUp keeps firing (Iris). Undo: PVCs cannot shrink — a git revert to 20Gi would make the Flux apply fail; the extra 20Gi (~60Gi raw of ~1.25TB free) is the accepted cost, so the fix-forward is to keep 40Gi. Root cause (what fills /paperclip) stays open.
Approved at f529698a993a07bd5c04ac2ce14c0147d910e43f.
Checked: diff is one line, pvc.yaml 20Gi→40Gi. Premise: alert fires on a real drain (Iris REN-1134: 3.18 GB free at 13:25Z; PR reads 2.19 GB at 15:00Z); PR claims headroom only, not root cause — correct scope. StorageClass ceph-block (repo storageclass.yaml): allowVolumeExpansion true, controller-expand secret set, ext4 → ceph-csi online resize, no pod restart. Precedent: gitea data-gitea-0 grown 40→60Gi via Flux (REN-583), live 60Gi now. Live: paperclip-data Bound 20Gi, prune: disabled (does not block spec apply). Flux paperclip Ready at 60e4ba4d. CI green (manifests, scripts, pr-review).
What breaks if wrong: resize fails → PVC sits at FileSystemResizePending, volume stays 20Gi and writable; no outage. Who notices: KubePersistentVolumeFillingUp keeps firing (Iris). Undo: PVCs cannot shrink — a git revert to 20Gi would make the Flux apply fail; the extra 20Gi (~60Gi raw of ~1.25TB free) is the accepted cost, so the fix-forward is to keep 40Gi. Root cause (what fills /paperclip) stays open.
Ghost
merged commit fc535074c0 into master2026-10-07 15:50:12 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ghost
What this PR does
KubePersistentVolumeFillingUpis firing forpaperclip/paperclip-data. The 20Gi RWO volume is mounted at/paperclipand shared by every agent that runs inside the paperclip container. Only about 2.19 GB is free, and space has been dropping by several GB per day. This PR grows the PVC to 40Gi. Theceph-blockStorageClass supports online expansion, so the volume should not run out before someone can look at it. This buys time. It does not fix the root cause. We still don't know what is filling the volume, and the diag account can't see inside it (no exec). The manual steps below are still needed.Changes
kubernetes/apps/paperclip/pvc.yaml:resources.requests.storagegoes from20Gito40Gi, with a short comment that points to this issue.Verification
Checked live this run:
kubectl -n paperclip get pvc,pods -o wide:paperclip-datais Bound with 20Gi onceph-blockand is 12 days old.paperclip-b65cc7768-msnkfis 1/1 Running with 0 restarts, onk8s-1.kubelet_volume_stats_available_bytes{persistentvolumeclaim="paperclip-data"}currently returns2188918784(about 2.19 GB). Over the last 12h (hourly samples) it fell from about 4.76 GB to 2.19 GB, with a drop of about 0.8 GB between 14:00Z and 15:00Z. The issue's numbers are correct.kubectl get storageclass ceph-block -o jsonpath={.allowVolumeExpansion}returnstrue.status.ceph.capacityshows about 1.25 TB available out of about 2.88 TB raw. Adding 20Gi (about 60Gi raw at 3× replication) is easily covered.After merge and Flux reconcile, check:
kubectl -n paperclip get pvc paperclip-datashould show CAPACITY40Gi. Look atkubectl -n paperclip describe pvc paperclip-dataforFileSystemResizeSuccessful. ceph-csi resizes the filesystem online, so no pod restart should be needed.kubelet_volume_stats_available_bytesshould rise by about 20 GB, and the alert should clear once the linear prediction stops pointing at zero within 4 days. If the current write rate continues (about 2.5–5 GB/day), the alert will come back in about a week. That is why the cleanup below matters.Required manual follow-up (root cause)
kubectl -n paperclip exec deploy/paperclip -- du -xh -d2 /paperclip | sort -h | tail -20/paperclip. Don't touch anything a run that is still active is using.Notes
kustomize.toolkit.fluxcd.io/prune: disabled. That only blocks deletion, so Flux still applies the spec change. Kubernetes PVCs can only grow, so going back to 20Gi later is not possible.kubectl -n paperclip logs deploy/paperclip --tail=60shows a tight retry loop, a few times a minute:sandbox callback bridge failed to write 504 backstop for e8395160-... (attempt n/3): ... sandboxes.agents.x-k8s.io "pc-01m4ak2swg1mxg6tzf1vy4454g" not found. This goes to container stdout, not to the PVC, so it is probably not why the volume is filling. It does look like an app bug: the bridge keeps retrying against a Sandbox that has already been deleted. The paperclip owner should look at it. The same log lines also show an API deprecation warning:agents.x-k8s.io/v1alpha1 Sandbox is deprecated; use v1beta1.Closes #1184
Approved at
f529698a99.Checked: diff is one line, pvc.yaml 20Gi→40Gi. Premise: alert fires on a real drain (Iris REN-1134: 3.18 GB free at 13:25Z; PR reads 2.19 GB at 15:00Z); PR claims headroom only, not root cause — correct scope. StorageClass ceph-block (repo storageclass.yaml): allowVolumeExpansion true, controller-expand secret set, ext4 → ceph-csi online resize, no pod restart. Precedent: gitea data-gitea-0 grown 40→60Gi via Flux (REN-583), live 60Gi now. Live: paperclip-data Bound 20Gi, prune: disabled (does not block spec apply). Flux paperclip Ready at
60e4ba4d. CI green (manifests, scripts, pr-review).What breaks if wrong: resize fails → PVC sits at FileSystemResizePending, volume stays 20Gi and writable; no outage. Who notices: KubePersistentVolumeFillingUp keeps firing (Iris). Undo: PVCs cannot shrink — a git revert to 20Gi would make the Flux apply fail; the extra 20Gi (~60Gi raw of ~1.25TB free) is the accepted cost, so the fix-forward is to keep 40Gi. Root cause (what fills /paperclip) stays open.