Full fix for #336 (gitea data-gitea-0 hit 100% and gitea began failing writes: "database or disk is full"). Supersedes #349 (which targeted tag count — the wrong layer).
What actually happened
The registry held 8.7 G of live, well-deduplicated images + 10.4 G of orphaned, unreferenced blobs (54% of the volume). Orphans come from image rebuilds overwriting tags (:latest, same-day dated tags). Gitea's default GC (@midnight, OLDER_THAN=24h) collected them only once a day, and a day's churn outran it. A manual GC reclaimed nothing because every orphan was <24 h old and skipped by the age cutoff.
The producer (root cause of the churn)
build-ci-images had a pipeline-level path trigger but no per-step filters, so any push to ci/images/** or secret-rotator/** rebuilt all five images. The recent run of secret-rotator commits each needlessly rebuilt the four ~1.3 G ci-* images, overwriting :latest and orphaning ~5 G per run. (shelly-manager building on every push+PR compounds it.)
Fix (two parts)
kubernetes/apps/gitea/statefulset.yaml — set [cron.cleanup_packages] to @every 6h, OLDER_THAN=4h (env encoding _0X2E_ verified in-pod). Orphans now get reclaimed same-day; applying this restarts gitea and its startup GC reclaims the existing ~10 G. 4h is well clear of any in-flight push.
.woodpecker/build-ci-images.yml — per-step when:path so each image rebuilds only when its own source (or the pipeline file) changed, or on manual. Kills the needless 4×1.3 G rebuilds.
Emergency relief already applied (out of band)
PVC expanded 20Gi → 40Gi live (ceph online resize) to clear the 0-bytes-free outage without a risky restart — gitea recovered immediately (now 50% used). Documented on the volumeClaimTemplate (its value stays 20Gi because that field is immutable on a live StatefulSet).
Follow-up (separate, not in this PR)
shelly-manager/vsftpd/authentik-samba-bridge push a kept image on every push and pull_request in their own repos — PR builds probably shouldn't push. Worth trimming there to cut churn further.
Full fix for #336 (gitea `data-gitea-0` hit 100% and gitea began failing writes: *"database or disk is full"*). Supersedes #349 (which targeted tag count — the wrong layer).
## What actually happened
The registry held **8.7 G of live, well-deduplicated images + 10.4 G of orphaned, unreferenced blobs** (54% of the volume). Orphans come from image rebuilds overwriting tags (`:latest`, same-day dated tags). Gitea's default GC (`@midnight`, `OLDER_THAN=24h`) collected them only once a day, and a day's churn outran it. A manual GC reclaimed nothing because every orphan was <24 h old and skipped by the age cutoff.
## The producer (root cause of the churn)
`build-ci-images` had a **pipeline-level** path trigger but **no per-step filters**, so any push to `ci/images/**` or `secret-rotator/**` rebuilt **all five images**. The recent run of secret-rotator commits each needlessly rebuilt the four ~1.3 G `ci-*` images, overwriting `:latest` and orphaning ~5 G per run. (`shelly-manager` building on every push+PR compounds it.)
## Fix (two parts)
1. **`kubernetes/apps/gitea/statefulset.yaml`** — set `[cron.cleanup_packages]` to `@every 6h`, `OLDER_THAN=4h` (env encoding `_0X2E_` verified in-pod). Orphans now get reclaimed same-day; applying this restarts gitea and its startup GC reclaims the existing ~10 G. `4h` is well clear of any in-flight push.
2. **`.woodpecker/build-ci-images.yml`** — per-step `when:path` so each image rebuilds only when its own source (or the pipeline file) changed, or on manual. Kills the needless 4×1.3 G rebuilds.
## Emergency relief already applied (out of band)
PVC expanded **20Gi → 40Gi** live (ceph online resize) to clear the 0-bytes-free outage without a risky restart — gitea recovered immediately (now 50% used). Documented on the volumeClaimTemplate (its value stays 20Gi because that field is immutable on a live StatefulSet).
## Follow-up (separate, not in this PR)
`shelly-manager`/`vsftpd`/`authentik-samba-bridge` push a kept image on every push **and pull_request** in their own repos — PR builds probably shouldn't push. Worth trimming there to cut churn further.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
data-gitea-0 hit 100% and gitea started failing writes ("database or
disk is full"). Root cause was NOT tag accumulation — live images are
only 8.7G and well-deduplicated. It was 10.4G (54% of the volume) of
ORPHANED, unreferenced blobs: every time an image tag is overwritten
(the ci-* images and shelly-manager rebuild frequently, shelly-manager
on both cron and push), the previous multi-GB layer set is orphaned.
Gitea's default cleanup_packages GC runs only @midnight with
OLDER_THAN=24h, so a day's worth of orphans piled up faster than it
collected them and filled the disk. A manual GC run reclaimed nothing
because every orphan was <24h old and thus skipped by the age cutoff.
Set the GC to run @every 6h with OLDER_THAN=4h so orphans are reclaimed
same-day. 4h is well beyond any in-flight push, so a live upload is
never collected. Applying this restarts gitea and its RUN_AT_START GC
reclaims the existing ~10G backlog (usage drops to ~9G).
Emergency relief was already applied out-of-band: the PVC was expanded
20Gi -> 40Gi (ceph-block online resize) to clear the outage without a
risky restart at 0 bytes free; documented on the volumeClaimTemplate.
Supersedes #349 (which targeted tag count, the wrong layer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-ci-images had a pipeline-level path trigger but no per-step
filters, so any push touching ci/images/** or secret-rotator/** rebuilt
ALL five images. The recent burst of secret-rotator commits therefore
needlessly rebuilt the four ~1.3G ci-* images each time, re-pushing and
overwriting :latest and orphaning ~5G of registry layers per run — a
primary source of the orphaned-blob pileup that filled data-gitea-0.
Add per-step when:path so each build runs only when its own source (or
the pipeline file) changed, or on a manual run. A secret-rotator change
now rebuilds only secret-rotator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the live 20Gi->40Gi expansion in the manifest as source of truth.
volumeClaimTemplates is immutable on an existing StatefulSet, so landing
this needs a one-time `kubectl delete statefulset gitea -n apps
--cascade=orphan` (pod + PVC survive) for Flux to re-create the STS from
the updated template. Documented inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Landing sequence (the statefulset now requests 40Gi, matching the live PVC):
Because volumeClaimTemplates is immutable on a live StatefulSet, an in-place apply is rejected (verified: Forbidden: updates ... other than replicas/template/... are forbidden), and since apply is atomic it would also block the GC env change. So after merge:
kubectl delete statefulset gitea -n apps --cascade=orphan # pod + PVC keep running
flux reconcile kustomization apps --with-source # Flux re-creates the STS from the 40Gi template + GC env, re-adopts the pod
The pod is orphaned (not deleted) during the STS delete, so there's no gitea downtime beyond the normal env-triggered pod roll — which is also what triggers the startup GC that reclaims the ~10G backlog. I can run this on merge and watch the reclaim.
**Landing sequence** (the statefulset now requests 40Gi, matching the live PVC):
Because `volumeClaimTemplates` is immutable on a live StatefulSet, an in-place `apply` is rejected (verified: `Forbidden: updates ... other than replicas/template/... are forbidden`), and since apply is atomic it would also block the GC env change. So after merge:
```
kubectl delete statefulset gitea -n apps --cascade=orphan # pod + PVC keep running
flux reconcile kustomization apps --with-source # Flux re-creates the STS from the 40Gi template + GC env, re-adopts the pod
```
The pod is orphaned (not deleted) during the STS delete, so there's no gitea downtime beyond the normal env-triggered pod roll — which is also what triggers the startup GC that reclaims the ~10G backlog. I can run this on merge and watch the reclaim.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Full fix for #336 (gitea
data-gitea-0hit 100% and gitea began failing writes: "database or disk is full"). Supersedes #349 (which targeted tag count — the wrong layer).What actually happened
The registry held 8.7 G of live, well-deduplicated images + 10.4 G of orphaned, unreferenced blobs (54% of the volume). Orphans come from image rebuilds overwriting tags (
:latest, same-day dated tags). Gitea's default GC (@midnight,OLDER_THAN=24h) collected them only once a day, and a day's churn outran it. A manual GC reclaimed nothing because every orphan was <24 h old and skipped by the age cutoff.The producer (root cause of the churn)
build-ci-imageshad a pipeline-level path trigger but no per-step filters, so any push toci/images/**orsecret-rotator/**rebuilt all five images. The recent run of secret-rotator commits each needlessly rebuilt the four ~1.3 Gci-*images, overwriting:latestand orphaning ~5 G per run. (shelly-managerbuilding on every push+PR compounds it.)Fix (two parts)
kubernetes/apps/gitea/statefulset.yaml— set[cron.cleanup_packages]to@every 6h,OLDER_THAN=4h(env encoding_0X2E_verified in-pod). Orphans now get reclaimed same-day; applying this restarts gitea and its startup GC reclaims the existing ~10 G.4his well clear of any in-flight push..woodpecker/build-ci-images.yml— per-stepwhen:pathso each image rebuilds only when its own source (or the pipeline file) changed, or on manual. Kills the needless 4×1.3 G rebuilds.Emergency relief already applied (out of band)
PVC expanded 20Gi → 40Gi live (ceph online resize) to clear the 0-bytes-free outage without a risky restart — gitea recovered immediately (now 50% used). Documented on the volumeClaimTemplate (its value stays 20Gi because that field is immutable on a live StatefulSet).
Follow-up (separate, not in this PR)
shelly-manager/vsftpd/authentik-samba-bridgepush a kept image on every push and pull_request in their own repos — PR builds probably shouldn't push. Worth trimming there to cut churn further.🤖 Generated with Claude Code
data-gitea-0 hit 100% and gitea started failing writes ("database or disk is full"). Root cause was NOT tag accumulation — live images are only 8.7G and well-deduplicated. It was 10.4G (54% of the volume) of ORPHANED, unreferenced blobs: every time an image tag is overwritten (the ci-* images and shelly-manager rebuild frequently, shelly-manager on both cron and push), the previous multi-GB layer set is orphaned. Gitea's default cleanup_packages GC runs only @midnight with OLDER_THAN=24h, so a day's worth of orphans piled up faster than it collected them and filled the disk. A manual GC run reclaimed nothing because every orphan was <24h old and thus skipped by the age cutoff. Set the GC to run @every 6h with OLDER_THAN=4h so orphans are reclaimed same-day. 4h is well beyond any in-flight push, so a live upload is never collected. Applying this restarts gitea and its RUN_AT_START GC reclaims the existing ~10G backlog (usage drops to ~9G). Emergency relief was already applied out-of-band: the PVC was expanded 20Gi -> 40Gi (ceph-block online resize) to clear the outage without a risky restart at 0 bytes free; documented on the volumeClaimTemplate. Supersedes #349 (which targeted tag count, the wrong layer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Landing sequence (the statefulset now requests 40Gi, matching the live PVC):
Because
volumeClaimTemplatesis immutable on a live StatefulSet, an in-placeapplyis rejected (verified:Forbidden: updates ... other than replicas/template/... are forbidden), and since apply is atomic it would also block the GC env change. So after merge:The pod is orphaned (not deleted) during the STS delete, so there's no gitea downtime beyond the normal env-triggered pod roll — which is also what triggers the startup GC that reclaims the ~10G backlog. I can run this on merge and watch the reclaim.