fix(rotation): repin every rotator to a dated tag, not a digest #459

2026-07-30T17:27:08Z

claude-bot commented

2026-07-30 17:27:08 +00:00

Closes #458.

The breakage

11 of the 14 rotators were pinned to image digests that no longer exist. Verified by pulling each in-cluster, not by inspecting metadata — the registry API returns 401 for a PAT, so metadata checks are misleading here.

digest	pull	rotators
`159e5888`	NotFound	authentik, crowdsec, paperless, shelly-manager, woodpecker, home-assistant (×3)
`978fe53c`	NotFound	cloudflare, external-dns
`49bfddd8`	NotFound	netbird-ingress
`97c22bcf`	NotFound	pve-exporter
`b41cc364`	pulls	flux-git
`828bf486`	pulls	claude-oauth, review-bot

The control: the two survivors are exactly the rotators that run often (claude-oauth-secret-sync is daily). Every gone digest belongs to a quarterly one. So this isn't a broken registry — it's untagged manifests being garbage-collected as tags move on.

Why it hadn't surfaced

Every quarterly CronJob shows lastScheduleTime: <none>. They were created 2026-07-18 → 07-23, after the last quarterly fire on 07-01 — so none has ever executed. First fire is 2026-10-01, where ~10 would have ImagePullBackOff'd simultaneously.

That failure mode is the dangerous one: a rotation that never starts looks identical to nothing happening. No alert fires for a Job that was never created, credentials silently stop rotating, and the manifests keep asserting they do.

The fix

All 16 references (13 files — home-assistant has 3 CronJobs, review-bot 2) now use secret-rotator:2026-07-30: dated tag, no digest, no :latest, per owner directive and matching ci-ansible:2026-04-29 and today's alpine-jq / cni-plugins / alpine-openssl pins.

Verified one image serves all of them before repinning. Modes in use across these files:

cloudflare-roll gitea-token ha-token netbird-key proxmox-token random woodpecker-secret

and the image's usage string lists exactly those seven. Single binary — splitting the pin would serve no purpose.

Also repinned flux-git, claude-oauth and review-bot, whose digests still resolve today. They're pinned identically and would decay the same way; converting now avoids a second pass when they break.

Root cause, recorded in `kustomization.yaml`

The rationale lives there rather than repeated 16 times — including why digests decayed, which is the non-obvious part:

The build triggers on .woodpecker/build-ci-images.yml, so adding an unrelated image republishes secret-rotator and orphans the manifest every rotator was pinned to.

That's how these went stale without anyone touching the rotator source — and why "repin to the current digest" would have been a trap that re-armed itself on the next unrelated image addition.

Verification

kustomize build + server-side dry-run clean; all 16 rendered CronJobs resolve to the single tag with their modes intact.

Worth noting the pattern is now proven working end-to-end: the mosquitto-health rotator (merged in #455, same tag) was run on demand and completed the full chain — rotator → OpenBao (23c10b3b → bc30778e) → ESO sync → Reloader roll → broker back 1/1 with u'health' clients authenticating on the new password, zero manual steps.

🤖 Generated with Claude Code

Closes #458. ## The breakage **11 of the 14 rotators were pinned to image digests that no longer exist.** Verified by *pulling each in-cluster*, not by inspecting metadata — the registry API returns 401 for a PAT, so metadata checks are misleading here. | digest | pull | rotators | |---|---|---| | `159e5888` | **NotFound** | authentik, crowdsec, paperless, shelly-manager, woodpecker, home-assistant (×3) | | `978fe53c` | **NotFound** | cloudflare, external-dns | | `49bfddd8` | **NotFound** | netbird-ingress | | `97c22bcf` | **NotFound** | pve-exporter | | `b41cc364` | pulls | flux-git | | `828bf486` | pulls | claude-oauth, review-bot | **The control**: the two survivors are exactly the rotators that run *often* (`claude-oauth-secret-sync` is daily). Every gone digest belongs to a quarterly one. So this isn't a broken registry — it's untagged manifests being garbage-collected as tags move on. ## Why it hadn't surfaced Every quarterly CronJob shows `lastScheduleTime: <none>`. They were created **2026-07-18 → 07-23**, after the last quarterly fire on 07-01 — so **none has ever executed**. First fire is **2026-10-01**, where ~10 would have `ImagePullBackOff`'d simultaneously. That failure mode is the dangerous one: a rotation that never *starts* looks identical to nothing happening. No alert fires for a Job that was never created, credentials silently stop rotating, and the manifests keep asserting they do. ## The fix All 16 references (13 files — `home-assistant` has 3 CronJobs, `review-bot` 2) now use **`secret-rotator:2026-07-30`**: dated tag, no digest, no `:latest`, per owner directive and matching `ci-ansible:2026-04-29` and today's `alpine-jq` / `cni-plugins` / `alpine-openssl` pins. **Verified one image serves all of them** before repinning. Modes in use across these files: ``` cloudflare-roll gitea-token ha-token netbird-key proxmox-token random woodpecker-secret ``` and the image's usage string lists exactly those seven. Single binary — splitting the pin would serve no purpose. **Also repinned `flux-git`, `claude-oauth` and `review-bot`**, whose digests still resolve today. They're pinned identically and would decay the same way; converting now avoids a second pass when they break. ## Root cause, recorded in `kustomization.yaml` The rationale lives there rather than repeated 16 times — including *why* digests decayed, which is the non-obvious part: > The build triggers on `.woodpecker/build-ci-images.yml`, so adding an **unrelated** image republishes `secret-rotator` and orphans the manifest every rotator was pinned to. That's how these went stale without anyone touching the rotator source — and why "repin to the current digest" would have been a trap that re-armed itself on the next unrelated image addition. ## Verification `kustomize build` + server-side dry-run clean; all 16 rendered CronJobs resolve to the single tag with their modes intact. Worth noting the pattern is now proven working end-to-end: the `mosquitto-health` rotator (merged in #455, same tag) was run on demand and completed the full chain — rotator → OpenBao (`23c10b3b` → `bc30778e`) → ESO sync → Reloader roll → broker back `1/1` with `u'health'` clients authenticating on the new password, zero manual steps. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-30 17:27:09 +00:00

fix(rotation): repin every rotator to a dated tag, not a digest

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

717a12bc18

Closes the breakage in #458: 11 of the 14 rotators were pinned to image
digests that no longer exist in the registry. Verified by pulling each
rather than by inspecting metadata (the registry API returns 401 for a
PAT, so metadata checks mislead here):

  gone     159e5888  authentik, crowdsec-bouncer-key, paperless,
                     shelly-manager, woodpecker, home-assistant (x3)
  gone     978fe53c  cloudflare, external-dns
  gone     49bfddd8  netbird-ingress
  gone     97c22bcf  pve-exporter
  present  b41cc364  flux-git
  present  828bf486  claude-oauth, review-bot

The two survivors are exactly the rotators that run often; every gone
digest belongs to a quarterly one. Untagged manifests are collected once
:latest and the dated tags move past them.

Impact had this not been caught: every quarterly CronJob shows
lastScheduleTime <none> — they were created 2026-07-18..23, after the last
quarterly fire — so none has ever run. Their first fire is 2026-10-01,
where ~10 would have ImagePullBackOff'd at once. A rotation that never
starts looks identical to nothing happening, so credentials would have
silently stopped rotating while the manifests kept asserting they did.

All 16 references (13 files; home-assistant has 3 CronJobs, review-bot 2)
now use secret-rotator:2026-07-30 — dated tag, no digest, no :latest, per
owner directive and matching ci-ansible:2026-04-29 and the alpine-jq /
cni-plugins / alpine-openssl pins.

Also repinned flux-git, claude-oauth and review-bot, whose digests still
resolve today. They are pinned the same way and would decay identically;
converting them now avoids a second pass when they break.

Verified before repinning that one image serves all of them: the modes in
use across these files are exactly

  cloudflare-roll gitea-token ha-token netbird-key proxmox-token random
  woodpecker-secret

and the 2026-07-30 image's usage string lists exactly those seven. So this
is a single binary and splitting the pin would serve no purpose.

Rationale recorded in kustomization.yaml rather than repeated 16 times,
including WHY digests decayed: the build triggers on
.woodpecker/build-ci-images.yml, so adding an unrelated image republishes
secret-rotator and orphans the manifest every rotator was pinned to. That
is how these went stale without anyone touching the rotator source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-07-30 17:30:29 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 717a12bc18e69aa35e5e3618d7c49b76ee5b9927.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Mechanical repin of all 16 secret-rotator image references (13 files, 16 CronJob/initContainer entries) from @sha256:... digests to the single dated tag secret-rotator:2026-07-30, plus a consolidated rationale comment in kustomization.yaml. This is a config-only change (no manifest logic, RBAC, or args touched) and I verified it thoroughly against the checkout — it's correct and complete.

No blocking issues.

🟡 Suggestions

Nothing to codify the "no digests" rule going forward. The root cause here (kubernetes/infrastructure/openbao/rotation/kustomization.yaml:8-25) is that a shared pipeline file (.woodpecker/build-ci-images.yml) causes secret-rotator to republish on unrelated changes, silently orphaning any digest a manifest was pinned to. That's well diagnosed and the comment explains it, but there's nothing to stop a future rotator file from reintroducing a @sha256: pin (e.g. copy-pasted from an older example or another part of the repo). A cheap guard would be a grep -rL '@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml step somewhere in CI, or even just a one-line note in the comment telling future authors to grep for @sha256 before adding a new rotator. Not required for this PR, just a durability idea.

Tests

No automated test exists for this directory (it's GitOps-deployed via Flux/kustomize, not covered by the Woodpecker CI pipelines I checked — deploy-infrastructure.yml only covers opentofu/**). The PR body describes manual verification (kustomize build + server-side dry-run, plus an in-cluster pull test of the new tag and an end-to-end run of the mosquitto-health rotator using the same tag in PR #455). That's reasonable for a change of this kind, and I independently confirmed the more checkable claims from the checkout:

All 16 image references across the 13 changed rotator files now point at gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30; a repo-wide grep found zero remaining secret-rotator@sha256:... references and zero leftover references to any of the six old digests. Every file listed in kustomization.yaml's resources: is accounted for.
secret-rotator/src/main.rs:33-40 dispatches on exactly the seven mode strings the PR claims (random, ha-token, cloudflare-roll, netbird-key, proxmox-token, gitea-token, woodpecker-secret), confirming the "one binary, one tag, all modes" reasoning for not splitting the pin.
.woodpecker/build-ci-images.yml:110-132 confirms the decay mechanism described in the new kustomization.yaml comment: the build-secret-rotator step's own when.path includes .woodpecker/build-ci-images.yml itself, so any edit to that shared file (e.g. adding a step for an unrelated new image) retriggers the secret-rotator build/push and republishes :latest/:$TAG — which is exactly how the old digest pins went stale. The new dated-tag scheme doesn't have this problem: :2026-07-30 is a named tag, not an orphaned untagged manifest, so it isn't subject to the same GC.

I could not independently verify the in-cluster pull tests or the OpenBao/ESO/Reloader end-to-end run described in the PR body (no registry or cluster access from this checkout), but nothing in the diff is inconsistent with those claims.

Recommendation

Looks reasonable — no blocking issues; comment above is optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `717a12bc18e69aa35e5e3618d7c49b76ee5b9927`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Mechanical repin of all 16 `secret-rotator` image references (13 files, 16 CronJob/initContainer entries) from `@sha256:...` digests to the single dated tag `secret-rotator:2026-07-30`, plus a consolidated rationale comment in `kustomization.yaml`. This is a config-only change (no manifest logic, RBAC, or args touched) and I verified it thoroughly against the checkout — it's correct and complete. No blocking issues. ## 🟡 Suggestions - **Nothing to codify the "no digests" rule going forward.** The root cause here (`kubernetes/infrastructure/openbao/rotation/kustomization.yaml:8-25`) is that a shared pipeline file (`.woodpecker/build-ci-images.yml`) causes `secret-rotator` to republish on unrelated changes, silently orphaning any digest a manifest was pinned to. That's well diagnosed and the comment explains it, but there's nothing to stop a future rotator file from reintroducing a `@sha256:` pin (e.g. copy-pasted from an older example or another part of the repo). A cheap guard would be a `grep -rL '@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml` step somewhere in CI, or even just a one-line note in the comment telling future authors to grep for `@sha256` before adding a new rotator. Not required for this PR, just a durability idea. ## Tests No automated test exists for this directory (it's GitOps-deployed via Flux/kustomize, not covered by the Woodpecker CI pipelines I checked — `deploy-infrastructure.yml` only covers `opentofu/**`). The PR body describes manual verification (`kustomize build` + server-side dry-run, plus an in-cluster pull test of the new tag and an end-to-end run of the `mosquitto-health` rotator using the same tag in PR #455). That's reasonable for a change of this kind, and I independently confirmed the more checkable claims from the checkout: - All 16 image references across the 13 changed rotator files now point at `gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30`; a repo-wide grep found zero remaining `secret-rotator@sha256:...` references and zero leftover references to any of the six old digests. Every file listed in `kustomization.yaml`'s `resources:` is accounted for. - `secret-rotator/src/main.rs:33-40` dispatches on exactly the seven mode strings the PR claims (`random`, `ha-token`, `cloudflare-roll`, `netbird-key`, `proxmox-token`, `gitea-token`, `woodpecker-secret`), confirming the "one binary, one tag, all modes" reasoning for not splitting the pin. - `.woodpecker/build-ci-images.yml:110-132` confirms the decay mechanism described in the new `kustomization.yaml` comment: the `build-secret-rotator` step's own `when.path` includes `.woodpecker/build-ci-images.yml` itself, so any edit to that shared file (e.g. adding a step for an unrelated new image) retriggers the `secret-rotator` build/push and republishes `:latest`/`:$TAG` — which is exactly how the old digest pins went stale. The new dated-tag scheme doesn't have this problem: `:2026-07-30` is a named tag, not an orphaned untagged manifest, so it isn't subject to the same GC. I could not independently verify the in-cluster pull tests or the OpenBao/ESO/Reloader end-to-end run described in the PR body (no registry or cluster access from this checkout), but nothing in the diff is inconsistent with those claims. ## Recommendation Looks reasonable — no blocking issues; comment above is optional.

claude-bot referenced this issue from a commit

2026-07-30 17:31:32 +00:00

docs(rotation): tell future authors how to avoid reintroducing digest pins

claude-bot added 1 commit 2026-07-30 17:31:32 +00:00

docs(rotation): tell future authors how to avoid reintroducing digest pins

ci/woodpecker/pr/pr-review Pipeline was canceled

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

1cad57984c

Review point on #459. Nothing stops the next rotator from copy-pasting a
@sha256 pin from an older example, and the failure only surfaces months
later on the first scheduled run — the same silent, delayed failure this
PR just cleaned up 11 instances of.

Added two copy-pasteable checks to the kustomization header: a grep that
must return nothing before adding a rotator, and a pull test to run BEFORE
committing a tag bump. The second matters because a tag that was never
built fails exactly like the dead digests did, and just as quietly.

Not a CI gate, deliberately. Nothing in CI covers kubernetes/** at all
(deploy-infrastructure.yml only covers opentofu/**), so a real guard means
standing up a lint/validate pipeline for that tree — worth doing, but as
its own change rather than smuggled into a repin PR. Documented here so
the knowledge is not lost while that waits.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot added 1 commit 2026-07-30 17:32:11 +00:00

fix(rotation): make the documented digest-guard not flag its own docs

ci/woodpecker/pr/pr-review Pipeline was canceled

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

5035085426

Ran the check I had just documented and it returned kustomization.yaml —
because the comment explaining the rule contains the literal string it
greps for. A guard whose only hit is its own documentation trains people
to ignore it, which is worse than not having one.

Anchored on `image:` instead. Verified both directions: returns nothing
across this directory as it stands, and still flags a file containing a
real `image: ...@sha256:` pin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot added 1 commit 2026-07-30 17:33:06 +00:00

fix(rotation): make the documented guard actually pass

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

5d32065b8f

Second attempt, and the first one was wrong for a subtler reason than I
fixed. Anchoring on `image:` did not help: this file documents the
pattern, so the comment text `image:.*@sha256` matches the regex
`image:.*@sha256`. The guard kept reporting its own documentation.

Added the `| grep -v kustomization.yaml` filter and this time ran the
command exactly as written rather than a paraphrase of it. Verified both
directions: no output across this directory as it stands, and it still
reports a file carrying a real `image: ...@sha256:` pin.

Lesson worth the two commits: a documented check has to be executed in the
form it is documented in. I had verified a variant twice and shipped a
guard whose only hit was itself — which trains people to ignore it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-07-30 17:35:24 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 5d32065b8ffbabbddca167220f5529a8443643dd.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR repins all 16 secret-rotator image references across 13 CronJob manifests (home-assistant ×3, review-bot ×2) from @sha256:... digests to the single dated tag secret-rotator:2026-07-30, and adds a detailed rationale to kustomization.yaml explaining why digest pins decay here. I verified the mechanics against the checkout rather than taking the PR body at face value: the grep guard in the new comment actually produces the claimed empty output, the seven rotator modes referenced in the manifests exactly match secret-rotator's usage string in secret-rotator/src/main.rs, and the claimed digest-decay mechanism is consistent with how .woodpecker/build-ci-images.yml's per-step when.path triggers are structured (every step, including build-secret-rotator, retriggers on any edit to that shared pipeline file). No blocking issues found.

🟡 Suggestions

The new "no digest pins" convention (kustomization.yaml:32-35) is only enforced by a grep a human is trusted to run before committing. Nothing in .woodpecker/ runs it in CI, so a future PR (or an unrelated automated edit) could silently reintroduce a @sha256 pin — the exact failure mode this PR just spent an incident fixing — and nothing would catch it before the next quarterly CronJob fire. Worth a follow-up: wire the documented grep into a CI step (e.g. alongside a kustomize build validation, which doesn't appear to exist in .woodpecker/ either) so this is enforced rather than relied on by convention.
The operational claims in the PR body (registry pulls returning NotFound, lastScheduleTime: <none>, the mosquitto-health end-to-end rotation test) are cluster-state facts I can't verify from this checkout. They're plausible and consistent with everything checkable in-repo, but they rest on trusting the author's out-of-band verification — noting this only so the reviewer knows it wasn't independently re-verified here.

Tests

No test suite applies to this change — it's image-tag references in Kubernetes manifests plus comments. The PR body states kustomize build + a server-side dry-run were run and all 16 rendered CronJobs resolve correctly; I could not re-run that here but the diff mechanically supports it (every touched line is a straight @sha256:... → :2026-07-30 swap with args/mode untouched). The self-checking grep in kustomization.yaml does currently pass against the tree (verified: grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml | grep -v kustomization.yaml returns nothing).

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `5d32065b8ffbabbddca167220f5529a8443643dd`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR repins all 16 `secret-rotator` image references across 13 CronJob manifests (home-assistant ×3, review-bot ×2) from `@sha256:...` digests to the single dated tag `secret-rotator:2026-07-30`, and adds a detailed rationale to `kustomization.yaml` explaining why digest pins decay here. I verified the mechanics against the checkout rather than taking the PR body at face value: the grep guard in the new comment actually produces the claimed empty output, the seven rotator modes referenced in the manifests exactly match `secret-rotator`'s usage string in `secret-rotator/src/main.rs`, and the claimed digest-decay mechanism is consistent with how `.woodpecker/build-ci-images.yml`'s per-step `when.path` triggers are structured (every step, including `build-secret-rotator`, retriggers on any edit to that shared pipeline file). No blocking issues found. ## 🟡 Suggestions - The new "no digest pins" convention (`kustomization.yaml:32-35`) is only enforced by a grep a human is trusted to run before committing. Nothing in `.woodpecker/` runs it in CI, so a future PR (or an unrelated automated edit) could silently reintroduce a `@sha256` pin — the exact failure mode this PR just spent an incident fixing — and nothing would catch it before the next quarterly CronJob fire. Worth a follow-up: wire the documented grep into a CI step (e.g. alongside a `kustomize build` validation, which doesn't appear to exist in `.woodpecker/` either) so this is enforced rather than relied on by convention. - The operational claims in the PR body (registry pulls returning NotFound, `lastScheduleTime: <none>`, the `mosquitto-health` end-to-end rotation test) are cluster-state facts I can't verify from this checkout. They're plausible and consistent with everything checkable in-repo, but they rest on trusting the author's out-of-band verification — noting this only so the reviewer knows it wasn't independently re-verified here. ## Tests No test suite applies to this change — it's image-tag references in Kubernetes manifests plus comments. The PR body states `kustomize build` + a server-side dry-run were run and all 16 rendered CronJobs resolve correctly; I could not re-run that here but the diff mechanically supports it (every touched line is a straight `@sha256:...` → `:2026-07-30` swap with args/mode untouched). The self-checking grep in `kustomization.yaml` does currently pass against the tree (verified: `grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml | grep -v kustomization.yaml` returns nothing). ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 971ea6dc09 into master

2026-07-30 17:44:23 +00:00

k.renner referenced this issue from a commit

2026-07-30 17:44:24 +00:00

Merge pull request 'fix(rotation): repin every rotator to a dated tag, not a digest' (#459) from rotator-repin into master

claude-bot referenced this issue from a commit

2026-07-30 17:56:07 +00:00

ci: validate the GitOps tree and pipeline configs on every PR

claude-bot referenced this pull request

2026-07-30 17:56:27 +00:00

ci: validate the GitOps tree and pipeline configs on every PR #460

Sign in to join this conversation.