fix(external-dns): dedicated Cloudflare token + reload on rotation #311

2026-07-19T09:41:30Z

claude-bot commented

2026-07-19 09:41:30 +00:00

Follow-up to the Caddy-token audit. The Caddy token's "unknown external consumer" was external-dns, in this cluster.

What the audit found

Commit 1f40366c records the 4-zone Caddy token as "shared with something outside this repo", and the Cloudflare rotation job pins token-id specifically to exclude it "so Caddy is never rolled". That premise was wrong.

Mapping each in-cluster Secret to a Cloudflare token ID via /user/tokens/verify:

Secret	Cloudflare token
`apps/cloudflare-api-token`	`f40570b2` = Caddy
`cert-manager/cloudflare-api-token`	`af708481` = Kubernetes Cert-Manager

Confirmed independently by call frequency: the Caddy token's last_used_on advanced every ~60s — external-dns's sync interval.

Comparing secret values (as an earlier sweep did) can only show that two secrets differ; it cannot show which Cloudflare token a value is. Only /user/tokens/verify answers that.

Cutover (already applied)

Minted Kubernetes External-DNS — Zone Read + DNS Write, renner.uno only, mirroring the cert-manager token's shape rather than Caddy's (which had DNS Write and no Zone Read). Verified scope: reads renner.uno, denied on the other zones.

Stored at OpenBao homelab/apps/external-dns, synced via ESO, external-dns restarted and reports All records are already up to date.

Traffic moved cleanly:

Caddy:        09:29:13 -> frozen for 11.5 min (zero calls since cutover)
External-DNS: 09:33:42 -> 09:34:42 -> 09:35:41 -> ... -> 09:39:45  (~60s)

The Caddy token now has zero callers and can be revoked, dropping write access to konstantin-renner.de, nerdcave.eu and sarah-renner.de. Revocation is deliberately not part of this PR.

What this PR changes

Only the Reloader annotation. external-dns reads CF_API_TOKEN as a startup env var, so rotating the Secret alone left the old token live in the running pod — during this cutover the restart had to be done by hand, and a scheduled rotation would have silently kept using the revoked token.

Placement is deliberate: deploymentAnnotations, not podAnnotations. Reloader has only ever been observed acting on workload-level annotations in this cluster.

Two follow-ups (not in this PR)

authentik-server, authentik-worker and crowdsec-lapi carry the Reloader annotation only at pod-template level. Reloader's logs show it acting on every workload-level-annotated resource and never on those three — but their secrets may simply not have changed yet, so this is unconfirmed, not a proven defect. A throwaway Deployment + Secret would settle it definitively.
apps/external-dns has no rotation automation, unlike its cert-manager sibling. Now that it holds a dedicated token, it is a straightforward Class-A/B candidate.

🤖 Generated with Claude Code

Follow-up to the Caddy-token audit. **The Caddy token's "unknown external consumer" was `external-dns`, in this cluster.** ## What the audit found Commit `1f40366c` records the 4-zone `Caddy` token as *"shared with something outside this repo"*, and the Cloudflare rotation job pins `token-id` specifically to exclude it **"so Caddy is never rolled"**. That premise was wrong. Mapping each in-cluster Secret to a Cloudflare token **ID** via `/user/tokens/verify`: | Secret | Cloudflare token | |---|---| | `apps/cloudflare-api-token` | `f40570b2` = **Caddy** | | `cert-manager/cloudflare-api-token` | `af708481` = Kubernetes Cert-Manager | Confirmed independently by call frequency: the Caddy token's `last_used_on` advanced every ~60s — external-dns's sync interval. Comparing secret *values* (as an earlier sweep did) can only show that two secrets differ; it cannot show *which* Cloudflare token a value is. Only `/user/tokens/verify` answers that. ## Cutover (already applied) Minted `Kubernetes External-DNS` — Zone Read + DNS Write, **`renner.uno` only**, mirroring the cert-manager token's shape rather than Caddy's (which had DNS Write and no Zone Read). Verified scope: reads `renner.uno`, **denied** on the other zones. Stored at OpenBao `homelab/apps/external-dns`, synced via ESO, external-dns restarted and reports `All records are already up to date`. Traffic moved cleanly: ``` Caddy: 09:29:13 -> frozen for 11.5 min (zero calls since cutover) External-DNS: 09:33:42 -> 09:34:42 -> 09:35:41 -> ... -> 09:39:45 (~60s) ``` The `Caddy` token now has **zero** callers and can be revoked, dropping write access to `konstantin-renner.de`, `nerdcave.eu` and `sarah-renner.de`. Revocation is deliberately **not** part of this PR. ## What this PR changes Only the Reloader annotation. external-dns reads `CF_API_TOKEN` as a startup env var, so rotating the Secret alone left the old token live in the running pod — during this cutover the restart had to be done by hand, and a scheduled rotation would have silently kept using the revoked token. Placement is deliberate: `deploymentAnnotations`, not `podAnnotations`. Reloader has only ever been observed acting on workload-level annotations in this cluster. ## Two follow-ups (not in this PR) 1. **`authentik-server`, `authentik-worker` and `crowdsec-lapi` carry the Reloader annotation only at pod-template level.** Reloader's logs show it acting on every workload-level-annotated resource and never on those three — but their secrets may simply not have changed yet, so this is **unconfirmed**, not a proven defect. A throwaway Deployment + Secret would settle it definitively. 2. **`apps/external-dns` has no rotation automation**, unlike its cert-manager sibling. Now that it holds a dedicated token, it is a straightforward Class-A/B candidate. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-19 09:41:30 +00:00

fix(external-dns): reload on Cloudflare token change 661af32aea

external-dns was moved off the shared 4-zone "Caddy" Cloudflare token onto a
dedicated single-zone token (Kubernetes External-DNS: Zone Read + DNS Write on
renner.uno only). Rotating that Secret alone is not enough: external-dns reads
CF_API_TOKEN as an env var at startup, so it kept serving with the old token
until the pod happened to restart. During this cutover that restart had to be
done by hand.

Add the Reloader annotation via deploymentAnnotations so a future rotation
restarts the Deployment automatically.

Placement matters: the annotation goes on the Deployment, not the pod template.
Reloader has only ever been observed acting on workload-level annotations in
this cluster (woodpecker, paperless, netbird-ingress, pve-exporter); the chart
exposes deploymentAnnotations for exactly this.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit b86d879f86 into master

2026-07-19 12:16:54 +00:00

k.renner referenced this issue from a commit

2026-07-19 12:16:55 +00:00

Merge pull request 'fix(external-dns): dedicated Cloudflare token + reload on rotation' (#311) from feat/external-dns-dedicated-token into master

claude-bot commented

2026-07-19 12:19:46 +00:00

Correction: follow-up 1 is refuted — no defect

The PR description flagged authentik-server, authentik-worker and crowdsec-lapi as possibly not reloading because their Reloader annotation sits at pod-template level. That suspicion is wrong. Tested rather than assumed.

A/B test: two identical Deployments consuming one Secret, differing only in annotation placement.

t+10s   annot-on-podtemplate=gen2   annot-on-workload=gen2

reloader: Changes detected in 'test-secret' ...; updated 'annot-on-podtemplate' of type 'Deployment'
reloader: Changes detected in 'test-secret' ...; updated 'annot-on-workload'    of type 'Deployment'

Reloader honours both placements and names each in its log. The reason it had never acted on those three is the innocent one: their secrets have not changed yet.

So those three workloads are correctly configured and the rotation fleet is unaffected. The code comment claiming the annotation must sit on the Deployment has been corrected in a follow-up commit — deploymentAnnotations is kept only to match the prevailing idiom here, not because pod-template annotations are ineffective.

Follow-up 2 (no rotation automation for apps/external-dns) still stands. Test namespace has been cleaned up.

## Correction: follow-up 1 is refuted — no defect The PR description flagged `authentik-server`, `authentik-worker` and `crowdsec-lapi` as *possibly* not reloading because their Reloader annotation sits at pod-template level. **That suspicion is wrong.** Tested rather than assumed. A/B test: two identical Deployments consuming one Secret, differing only in annotation placement. ``` t+10s annot-on-podtemplate=gen2 annot-on-workload=gen2 reloader: Changes detected in 'test-secret' ...; updated 'annot-on-podtemplate' of type 'Deployment' reloader: Changes detected in 'test-secret' ...; updated 'annot-on-workload' of type 'Deployment' ``` Reloader honours **both** placements and names each in its log. The reason it had never acted on those three is the innocent one: their secrets have not changed yet. So those three workloads are correctly configured and the rotation fleet is unaffected. The code comment claiming the annotation *must* sit on the Deployment has been corrected in a follow-up commit — `deploymentAnnotations` is kept only to match the prevailing idiom here, not because pod-template annotations are ineffective. Follow-up 2 (no rotation automation for `apps/external-dns`) still stands. Test namespace has been cleaned up.

claude-bot referenced this pull request

2026-07-19 12:25:30 +00:00

docs(external-dns): correct the Reloader annotation-placement claim #312

claude-bot referenced this issue from a commit

2026-07-19 12:31:07 +00:00

feat(external-dns): automated Cloudflare token rotation

claude-bot referenced this pull request

2026-07-19 12:31:30 +00:00

feat(external-dns): automated Cloudflare token rotation #314

Sign in to join this conversation.