feat(monitoring): woodpecker secret-rotation convergence alerts #298

2026-07-18T18:33:18Z

claude-bot commented

2026-07-18 18:33:18 +00:00

Follow-up to the rotation PR #297, closing the H2 gap the adversarial review flagged: the rotation Job is fire-and-forget, so an ESO/Reloader/pod-reschedule failure downstream leaves a green Job with no alert. ESO metrics aren't scraped in this cluster, so these are kube-state-metrics based.

New PrometheusRule woodpecker-rotation (centralized in monitoring-rules, release: kube-prometheus-stack):

Alert	Sev	Fires when
WoodpeckerServerNotReady	critical	server StatefulSet 0 ready for 20m (CI fully down)
WoodpeckerAgentNotReady	warning	agent Deployment 0 available for 20m
WoodpeckerSecretRotationJobFailed	warning	a rotation Job failed (clearer than generic KubeJobFailed; e.g. missing OpenBao bootstrap)
WoodpeckerSecretRotationStale	warning	quarterly CronJob hasn't succeeded in >100d (absent-until-first-success → no early false-fire)

Independent of #297 — the readiness alerts are useful immediately; the Job/CronJob series simply stay empty until #297 merges. for: 20m is sized comfortably past the 10m ESO refresh + Reloader restart cycle so normal rotations don't trip it. Verified: kubectl kustomize builds clean, yamllint clean.

Follow-up to the rotation PR #297, closing the **H2** gap the adversarial review flagged: the rotation Job is fire-and-forget, so an ESO/Reloader/pod-reschedule failure downstream leaves a *green* Job with no alert. ESO metrics aren't scraped in this cluster, so these are kube-state-metrics based. New PrometheusRule `woodpecker-rotation` (centralized in monitoring-rules, `release: kube-prometheus-stack`): | Alert | Sev | Fires when | |---|---|---| | WoodpeckerServerNotReady | critical | server StatefulSet 0 ready for 20m (CI fully down) | | WoodpeckerAgentNotReady | warning | agent Deployment 0 available for 20m | | WoodpeckerSecretRotationJobFailed | warning | a rotation Job failed (clearer than generic KubeJobFailed; e.g. missing OpenBao bootstrap) | | WoodpeckerSecretRotationStale | warning | quarterly CronJob hasn't succeeded in >100d (absent-until-first-success → no early false-fire) | **Independent of #297** — the readiness alerts are useful immediately; the Job/CronJob series simply stay empty until #297 merges. `for: 20m` is sized comfortably past the 10m ESO refresh + Reloader restart cycle so normal rotations don't trip it. Verified: `kubectl kustomize` builds clean, yamllint clean.

claude-bot added 1 commit 2026-07-18 18:33:18 +00:00

feat(monitoring): woodpecker secret-rotation convergence alerts a3dc001361

Closes the H2 gap from the rotation PR (#297) adversarial review: the rotation Job is fire-and-forget, so a downstream failure (ESO/Reloader/pod-reschedule) yields a green Job with no signal. ESO metrics aren't scraped here, so these are kube-state-metrics based:
- WoodpeckerServerNotReady (critical): server StatefulSet 0 ready for 20m.
- WoodpeckerAgentNotReady (warning): agent Deployment 0 available for 20m.
- WoodpeckerSecretRotationJobFailed (warning): named signal for a failed rotation Job (e.g. missing OpenBao bootstrap).
- WoodpeckerSecretRotationStale (warning): quarterly CronJob hasn't succeeded in >100d; absent-until-first-success so it can't false-fire early.
Independent of #297: readiness alerts are useful immediately; the Job/CronJob series stay empty until that PR merges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ

k.renner merged commit 1f67d1d081 into master

2026-07-18 18:37:55 +00:00

k.renner referenced this issue from a commit

2026-07-18 18:37:57 +00:00

Merge pull request 'feat(monitoring): woodpecker secret-rotation convergence alerts' (#298) from feat/woodpecker-rotation-alerts into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#298