KubeJobFailed fired for job/claude-oauth-secret-sync-29785100 in openbao because the
CronJob still pointed at woodpecker.apps.svc.cluster.local:8000, a Service that no longer
exists: the Woodpecker apps → woodpecker namespace migration
(kubernetes/apps/woodpecker/MIGRATION-namespace.md) completed its cutover, but the two
openbao rotators' WOODPECKER_URL env vars were left behind. The container fails DNS
resolution and exits 1 within a second, twice, hitting BackoffLimitExceeded. This changes
both rotators' WOODPECKER_URL to woodpecker.woodpecker.svc.cluster.local:8000 (plus the
now-stale header comments that documented the old address). The issue's diagnosis and both
suggested fix locations were correct and are confirmed by live evidence.
Changes
kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml — WOODPECKER_URL → http://woodpecker.woodpecker.svc.cluster.local:8000; header comment updated to name the
new address and point at the migration doc. This is the CronJob that alerted.
kubernetes/infrastructure/openbao/rotation/review-bot.yaml — identical one-line fix for
the same stale URL. Not alerting yet only because it runs quarterly; it would fail the same
way on its next run. The gitea.apps.svc:3000 hop in the same file is left alone — gitea
really is still in apps (verified).
Verification
Verified live this run:
./.ci-diag/kubectl -n apps get svc → 7 services (external-dns, gitea, gatus, file, ical-proxy, pve, external-dns-unifi); no woodpecker, so the old name cannot resolve.
./.ci-diag/kubectl -n woodpecker get svc -o wide → woodpecker ClusterIP 10.99.97.9,
ports 8000/TCP,9000/TCP, age 6h16m — the new target exists.
./.ci-diag/kubectl -n openbao logs claude-oauth-secret-sync-29785100-qvcn2 --tail=20 → error: io: failed to lookup address information: Name does not resolve (read from the live
pod, which still exists — not from the issue's quote). A DNS failure, not auth and not netpol.
./.ci-diag/kubectl -n openbao get jobs → ...29783660 Complete (25h ago), ...29785100 Failed (81m ago). The last success predates the ~6h-old cutover and the first
failure follows it — the timeline matches the migration, not a transient.
./.ci-diag/kubectl -n woodpecker get cnp woodpecker-allow-rotators -o jsonpath='{.spec}' →
already admits both review-bot and claude-oauth rotators from openbao to the server on :8000. The egress halves in network-policies/openbao-ns.yaml already target k8s:io.kubernetes.pod.namespace: woodpecker. So no policy change is needed — the URL was
the only thing left pointing at apps.
./.ci-diag/kubectl apply --dry-run=client -f <both files> → both CronJobs configured (dry run); the only errors are expected RBAC denials reading ServiceAccounts.
After Flux reconciles, confirm without waiting for 04:20 Europe/Berlin:
Expect a clean exit, not the DNS error. KubeJobFailed for claude-oauth-secret-sync-29785100 should resolve once the failed job is deleted (per the
CronJob's own runbook note) — the alert reflects the existing failed object and will not clear
on its own just because the manifest changed.
Notes
Not a silenced alert. The failing job is fixed at its cause; no rule, threshold, or
inhibition was touched.
Follow-up, deliberately not fixed here (separate defect, different alert family, and
larger than a surgical change): kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml
has four alert expressions still selecting namespace="apps" (lines 24, 38, 53, 68), and
line 32's description still cites woodpecker.apps.svc:9000. I confirmed these are now dead
rather than merely cosmetic: a Prometheus query for kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"}
returns an empty vector, and kubectl -n woodpecker get statefulset,deploy shows woodpecker-server is now a Deployment in woodpecker — so WoodpeckerServerNotReady
is wrong on both namespace and resource kind and can never fire. That means Woodpecker being
down is currently unalerted. Worth its own issue; fixing it means rewriting four exprs and
their descriptions, and I'd rather that be reviewed on its own than smuggled into a
one-line DNS fix.
Image tags, pinned versions, and CI config were not touched. The issue contained no
instructions to act outside the edit scope.
## What this PR does
`KubeJobFailed` fired for `job/claude-oauth-secret-sync-29785100` in `openbao` because the
CronJob still pointed at `woodpecker.apps.svc.cluster.local:8000`, a Service that no longer
exists: the Woodpecker `apps` → `woodpecker` namespace migration
(`kubernetes/apps/woodpecker/MIGRATION-namespace.md`) completed its cutover, but the two
openbao rotators' `WOODPECKER_URL` env vars were left behind. The container fails DNS
resolution and exits 1 within a second, twice, hitting `BackoffLimitExceeded`. This changes
both rotators' `WOODPECKER_URL` to `woodpecker.woodpecker.svc.cluster.local:8000` (plus the
now-stale header comments that documented the old address). The issue's diagnosis and both
suggested fix locations were correct and are confirmed by live evidence.
## Changes
- `kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml` — `WOODPECKER_URL` →
`http://woodpecker.woodpecker.svc.cluster.local:8000`; header comment updated to name the
new address and point at the migration doc. This is the CronJob that alerted.
- `kubernetes/infrastructure/openbao/rotation/review-bot.yaml` — identical one-line fix for
the same stale URL. Not alerting yet only because it runs quarterly; it would fail the same
way on its next run. The `gitea.apps.svc:3000` hop in the same file is left alone — gitea
really is still in `apps` (verified).
## Verification
Verified live this run:
- `./.ci-diag/kubectl -n apps get svc` → 7 services (`external-dns`, `gitea`, `gatus`, `file`,
`ical-proxy`, `pve`, `external-dns-unifi`); **no `woodpecker`**, so the old name cannot resolve.
- `./.ci-diag/kubectl -n woodpecker get svc -o wide` → `woodpecker` ClusterIP `10.99.97.9`,
ports `8000/TCP,9000/TCP`, age `6h16m` — the new target exists.
- `./.ci-diag/kubectl -n openbao logs claude-oauth-secret-sync-29785100-qvcn2 --tail=20` →
`error: io: failed to lookup address information: Name does not resolve` (read from the live
pod, which still exists — not from the issue's quote). A DNS failure, not auth and not netpol.
- `./.ci-diag/kubectl -n openbao get jobs` → `...29783660 Complete` (25h ago),
`...29785100 Failed` (81m ago). The last success predates the ~6h-old cutover and the first
failure follows it — the timeline matches the migration, not a transient.
- `./.ci-diag/kubectl -n woodpecker get cnp woodpecker-allow-rotators -o jsonpath='{.spec}'` →
already admits both `review-bot` and `claude-oauth` rotators from `openbao` to the server on
`:8000`. The egress halves in `network-policies/openbao-ns.yaml` already target
`k8s:io.kubernetes.pod.namespace: woodpecker`. **So no policy change is needed — the URL was
the only thing left pointing at `apps`.**
- `./.ci-diag/kubectl apply --dry-run=client -f <both files>` → both CronJobs
`configured (dry run)`; the only errors are expected RBAC denials reading ServiceAccounts.
After Flux reconciles, confirm without waiting for 04:20 Europe/Berlin:
```sh
kubectl -n openbao get cronjob claude-oauth-secret-sync \
-o jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].env[?(@.name=="WOODPECKER_URL")].value}'
kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now
kubectl -n openbao logs job/claude-oauth-sync-now
```
Expect a clean exit, not the DNS error. `KubeJobFailed` for
`claude-oauth-secret-sync-29785100` should resolve once the failed job is deleted (per the
CronJob's own runbook note) — the alert reflects the existing failed object and will not clear
on its own just because the manifest changed.
## Notes
- **Not a silenced alert.** The failing job is fixed at its cause; no rule, threshold, or
inhibition was touched.
- **Follow-up, deliberately not fixed here** (separate defect, different alert family, and
larger than a surgical change): `kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml`
has four alert expressions still selecting `namespace="apps"` (lines 24, 38, 53, 68), and
line 32's description still cites `woodpecker.apps.svc:9000`. I confirmed these are now dead
rather than merely cosmetic: a Prometheus query for
`kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"}`
returns an empty vector, and `kubectl -n woodpecker get statefulset,deploy` shows
`woodpecker-server` is now a **Deployment** in `woodpecker` — so `WoodpeckerServerNotReady`
is wrong on both namespace and resource kind and can never fire. That means Woodpecker being
down is currently unalerted. Worth its own issue; fixing it means rewriting four exprs and
their descriptions, and I'd rather that be reviewed on its own than smuggled into a
one-line DNS fix.
- Image tags, pinned versions, and CI config were not touched. The issue contained no
instructions to act outside the edit scope.
Closes #510
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit d9f1d6cfc896e043e5b17280dde985cf1bc89237.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR fixes a DNS resolution failure in two OpenBao secret rotation CronJobs by updating their WOODPECKER_URL environment variable from woodpecker.apps.svc.cluster.local:8000 to woodpecker.woodpecker.svc.cluster.local:8000, reflecting the Woodpecker namespace migration that was already completed. The fix is minimal, surgical, and thoroughly verified.
🔴 Blocking
None. The changes are correct and directly address the reported issue.
🟡 Suggestions
Comment clarity — in both files, the unchanged portions of comments still reference woodpecker-allow-rotators (apps/*) notation. While the PR author verified the NetworkPolicy is already correctly configured in the woodpecker namespace, this notation may be unclear to future readers. This could be clarified in a follow-up, but is not a correctness issue — the policy itself (in kubernetes/apps/woodpecker/networkpolicy.yaml) is correctly scoped to the woodpecker server pod and already admits both rotators from openbao on :8000. Consider a follow-up PR to clarify the comment notation if needed.
🟢 Nits
None.
Tests
This is infrastructure code (Kubernetes manifests). The PR includes thorough manual verification:
Confirmed the old service no longer exists in the apps namespace
Confirmed the new service exists and is healthy in the woodpecker namespace
Verified DNS resolution errors in live pod logs
Confirmed network policies are already correctly configured
Verified kubectl apply --dry-run=client succeeds
The verification approach is appropriate. After Flux reconciles, the suggested post-merge validation step (creating a job from the CronJob and checking logs for clean exit instead of DNS errors) is a solid way to confirm the fix works in production.
Recommendation
Looks reasonable — no blocking issues; the suggestion above is optional and can be addressed in a follow-up.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `d9f1d6cfc896e043e5b17280dde985cf1bc89237`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR fixes a DNS resolution failure in two OpenBao secret rotation CronJobs by updating their `WOODPECKER_URL` environment variable from `woodpecker.apps.svc.cluster.local:8000` to `woodpecker.woodpecker.svc.cluster.local:8000`, reflecting the Woodpecker namespace migration that was already completed. The fix is minimal, surgical, and thoroughly verified.
## 🔴 Blocking
None. The changes are correct and directly address the reported issue.
## 🟡 Suggestions
**Comment clarity** — in both files, the unchanged portions of comments still reference `woodpecker-allow-rotators (apps/*)` notation. While the PR author verified the NetworkPolicy is already correctly configured in the `woodpecker` namespace, this notation may be unclear to future readers. This could be clarified in a follow-up, but is not a correctness issue — the policy itself (in `kubernetes/apps/woodpecker/networkpolicy.yaml`) is correctly scoped to the woodpecker server pod and already admits both rotators from openbao on `:8000`. Consider a follow-up PR to clarify the comment notation if needed.
## 🟢 Nits
None.
## Tests
This is infrastructure code (Kubernetes manifests). The PR includes thorough manual verification:
- Confirmed the old service no longer exists in the `apps` namespace
- Confirmed the new service exists and is healthy in the `woodpecker` namespace
- Verified DNS resolution errors in live pod logs
- Confirmed network policies are already correctly configured
- Verified `kubectl apply --dry-run=client` succeeds
The verification approach is appropriate. After Flux reconciles, the suggested post-merge validation step (creating a job from the CronJob and checking logs for clean exit instead of DNS errors) is a solid way to confirm the fix works in production.
## Recommendation
Looks reasonable — no blocking issues; the suggestion above is optional and can be addressed in a follow-up.
<!-- pr-review:sha=d9f1d6cfc896e043e5b17280dde985cf1bc89237 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
KubeJobFailedfired forjob/claude-oauth-secret-sync-29785100inopenbaobecause theCronJob still pointed at
woodpecker.apps.svc.cluster.local:8000, a Service that no longerexists: the Woodpecker
apps→woodpeckernamespace migration(
kubernetes/apps/woodpecker/MIGRATION-namespace.md) completed its cutover, but the twoopenbao rotators'
WOODPECKER_URLenv vars were left behind. The container fails DNSresolution and exits 1 within a second, twice, hitting
BackoffLimitExceeded. This changesboth rotators'
WOODPECKER_URLtowoodpecker.woodpecker.svc.cluster.local:8000(plus thenow-stale header comments that documented the old address). The issue's diagnosis and both
suggested fix locations were correct and are confirmed by live evidence.
Changes
kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml—WOODPECKER_URL→http://woodpecker.woodpecker.svc.cluster.local:8000; header comment updated to name thenew address and point at the migration doc. This is the CronJob that alerted.
kubernetes/infrastructure/openbao/rotation/review-bot.yaml— identical one-line fix forthe same stale URL. Not alerting yet only because it runs quarterly; it would fail the same
way on its next run. The
gitea.apps.svc:3000hop in the same file is left alone — giteareally is still in
apps(verified).Verification
Verified live this run:
./.ci-diag/kubectl -n apps get svc→ 7 services (external-dns,gitea,gatus,file,ical-proxy,pve,external-dns-unifi); nowoodpecker, so the old name cannot resolve../.ci-diag/kubectl -n woodpecker get svc -o wide→woodpeckerClusterIP10.99.97.9,ports
8000/TCP,9000/TCP, age6h16m— the new target exists../.ci-diag/kubectl -n openbao logs claude-oauth-secret-sync-29785100-qvcn2 --tail=20→error: io: failed to lookup address information: Name does not resolve(read from the livepod, which still exists — not from the issue's quote). A DNS failure, not auth and not netpol.
./.ci-diag/kubectl -n openbao get jobs→...29783660 Complete(25h ago),...29785100 Failed(81m ago). The last success predates the ~6h-old cutover and the firstfailure follows it — the timeline matches the migration, not a transient.
./.ci-diag/kubectl -n woodpecker get cnp woodpecker-allow-rotators -o jsonpath='{.spec}'→already admits both
review-botandclaude-oauthrotators fromopenbaoto the server on:8000. The egress halves innetwork-policies/openbao-ns.yamlalready targetk8s:io.kubernetes.pod.namespace: woodpecker. So no policy change is needed — the URL wasthe only thing left pointing at
apps../.ci-diag/kubectl apply --dry-run=client -f <both files>→ both CronJobsconfigured (dry run); the only errors are expected RBAC denials reading ServiceAccounts.After Flux reconciles, confirm without waiting for 04:20 Europe/Berlin:
Expect a clean exit, not the DNS error.
KubeJobFailedforclaude-oauth-secret-sync-29785100should resolve once the failed job is deleted (per theCronJob's own runbook note) — the alert reflects the existing failed object and will not clear
on its own just because the manifest changed.
Notes
inhibition was touched.
larger than a surgical change):
kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yamlhas four alert expressions still selecting
namespace="apps"(lines 24, 38, 53, 68), andline 32's description still cites
woodpecker.apps.svc:9000. I confirmed these are now deadrather than merely cosmetic: a Prometheus query for
kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"}returns an empty vector, and
kubectl -n woodpecker get statefulset,deployshowswoodpecker-serveris now a Deployment inwoodpecker— soWoodpeckerServerNotReadyis wrong on both namespace and resource kind and can never fire. That means Woodpecker being
down is currently unalerted. Worth its own issue; fixing it means rewriting four exprs and
their descriptions, and I'd rather that be reviewed on its own than smuggled into a
one-line DNS fix.
instructions to act outside the edit scope.
Closes #510
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commitd9f1d6cfc896e043e5b17280dde985cf1bc89237.Summary
This PR fixes a DNS resolution failure in two OpenBao secret rotation CronJobs by updating their
WOODPECKER_URLenvironment variable fromwoodpecker.apps.svc.cluster.local:8000towoodpecker.woodpecker.svc.cluster.local:8000, reflecting the Woodpecker namespace migration that was already completed. The fix is minimal, surgical, and thoroughly verified.🔴 Blocking
None. The changes are correct and directly address the reported issue.
🟡 Suggestions
Comment clarity — in both files, the unchanged portions of comments still reference
woodpecker-allow-rotators (apps/*)notation. While the PR author verified the NetworkPolicy is already correctly configured in thewoodpeckernamespace, this notation may be unclear to future readers. This could be clarified in a follow-up, but is not a correctness issue — the policy itself (inkubernetes/apps/woodpecker/networkpolicy.yaml) is correctly scoped to the woodpecker server pod and already admits both rotators from openbao on:8000. Consider a follow-up PR to clarify the comment notation if needed.🟢 Nits
None.
Tests
This is infrastructure code (Kubernetes manifests). The PR includes thorough manual verification:
appsnamespacewoodpeckernamespacekubectl apply --dry-run=clientsucceedsThe verification approach is appropriate. After Flux reconciles, the suggested post-merge validation step (creating a job from the CronJob and checking logs for clean exit instead of DNS errors) is a solid way to confirm the fix works in production.
Recommendation
Looks reasonable — no blocking issues; the suggestion above is optional and can be addressed in a follow-up.