feat(ci-agents): verified live diagnostics for the triage/fix agents (no more blind assessments) #253

2026-07-16T08:13:16Z

claude-bot commented

2026-07-16 08:13:16 +00:00

Fixes the agents, not the alerts. Both wrong assessments (#249 "pod is 0/1 and sealed" — it was Ready and unsealed; #250 endorsing three untested hypotheses while one kubectl logs showed the real 401 Unauthorized from the token revoked in #213) share one root cause: the agents diagnosed blind and filled the evidence gap with plausible narrative.

Two commits, two layers:

1. Honesty (`33eb659`)

Prompts forbid asserting live cluster state that wasn't observed: runtime claims must be attributed or labeled hypothesis, and the triage agent must explicitly flag when the gathered context does not confirm the alert's symptom (that mismatch was the actual diagnosis on #249 — updateStrategy: OnDelete).

2. Verification (`90aeaf1`)

The agents get read-only live diagnostics, so they can verify instead of hypothesize:

RBAC (kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml): ServiceAccount monitoring/ci-diagnostics, ClusterRole readable on workloads/events/pods/log/Flux/ESO/cert-manager/monitoring/ceph CRDs — Secrets deliberately excluded (findings land in public issues), plus a namespaced services/proxy Role for the Prometheus/Alertmanager HTTP APIs (both bare and :port resourceName forms).
Token plumbing (scripts/ci/make-diag-kubeconfig.sh): each run mints a 1-hour token via kubectl create token from the already-fetched admin kubeconfig — no new long-lived secret anywhere. Copies the static kubectl binary into the workspace for the jq-only ci-claude image, smoke-tests with the restricted identity, and degrades to snapshot-only mode (with mandatory "unverified" labeling) if the SA isn't reconciled yet.
Tool surface: agents gain exactly one extra allowlisted command, Bash(./.ci-diag/kubectl:*). Prefix matching forbids pipes/redirects/other binaries; RBAC forbids mutation and secret reads. Everything else (no web, no push from the triage agent, pipeline does all Gitea writes) is unchanged.
Prompts: verification is mandatory — "a hypothesis you could have settled with one read-only command but didn't is a defect in your output"; the fix agent must test the issue's claims before building on them; minimal-quote rule so pod logs never get dumped into public issues.

Residual risk, stated plainly

pods/log is readable and issues are public — a workload that logs something sensitive could leak via a quoted line. Mitigations: prompt minimal-quote rule + no-Secrets RBAC. Drop pods/log from the ClusterRole if you'd rather trade diagnostic power for zero log exposure.

Verified pre-merge

RBAC applies clean (kubectl apply --dry-run=server), monitoring kustomization builds, both pipelines pass woodpecker lint, script survives bash -n and its no-SA fallback path was executed for real.
Post-merge I'll run make-diag-kubeconfig.sh locally and confirm the restricted identity: can read pods/logs/proxy APIs, gets 403 on secrets and on writes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

Fixes the agents, not the alerts. Both wrong assessments (#249 "pod is 0/1 and sealed" — it was Ready and unsealed; #250 endorsing three untested hypotheses while one `kubectl logs` showed the real `401 Unauthorized` from the token revoked in #213) share one root cause: **the agents diagnosed blind and filled the evidence gap with plausible narrative.** Two commits, two layers: ## 1. Honesty (`33eb659`) Prompts forbid asserting live cluster state that wasn't observed: runtime claims must be attributed or labeled hypothesis, and the triage agent must explicitly flag when the gathered context does not confirm the alert's symptom (that mismatch was the actual diagnosis on #249 — `updateStrategy: OnDelete`). ## 2. Verification (`90aeaf1`) The agents get **read-only live diagnostics**, so they can verify instead of hypothesize: - **RBAC** (`kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml`): ServiceAccount `monitoring/ci-diagnostics`, ClusterRole readable on workloads/events/`pods/log`/Flux/ESO/cert-manager/monitoring/ceph CRDs — **Secrets deliberately excluded** (findings land in public issues), plus a namespaced `services/proxy` Role for the Prometheus/Alertmanager HTTP APIs (both bare and `:port` resourceName forms). - **Token plumbing** (`scripts/ci/make-diag-kubeconfig.sh`): each run mints a **1-hour token** via `kubectl create token` from the already-fetched admin kubeconfig — no new long-lived secret anywhere. Copies the static kubectl binary into the workspace for the jq-only ci-claude image, smoke-tests with the restricted identity, and **degrades to snapshot-only mode** (with mandatory "unverified" labeling) if the SA isn't reconciled yet. - **Tool surface**: agents gain exactly one extra allowlisted command, `Bash(./.ci-diag/kubectl:*)`. Prefix matching forbids pipes/redirects/other binaries; RBAC forbids mutation and secret reads. Everything else (no web, no push from the triage agent, pipeline does all Gitea writes) is unchanged. - **Prompts**: verification is mandatory — *"a hypothesis you could have settled with one read-only command but didn't is a defect in your output"*; the fix agent must test the issue's claims before building on them; minimal-quote rule so pod logs never get dumped into public issues. ## Residual risk, stated plainly `pods/log` is readable and issues are public — a workload that logs something sensitive could leak via a quoted line. Mitigations: prompt minimal-quote rule + no-Secrets RBAC. Drop `pods/log` from the ClusterRole if you'd rather trade diagnostic power for zero log exposure. ## Verified pre-merge - RBAC applies clean (`kubectl apply --dry-run=server`), monitoring kustomization builds, both pipelines pass `woodpecker lint`, script survives `bash -n` and its no-SA fallback path was executed for real. - Post-merge I'll run `make-diag-kubeconfig.sh` locally and confirm the restricted identity: can read pods/logs/proxy APIs, gets 403 on secrets and on writes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

claude-bot added 1 commit 2026-07-16 08:13:17 +00:00

fix(ci-agents): forbid asserting unverified cluster state in triage/fix prompts 33eb6592c6

The first issue-fix run on #249 asserted the openbao pod was '0/1 and
sealed' — it was Ready and unsealed the whole time; the alert fired
because the chart's updateStrategy: OnDelete holds the 2.5.5→2.6.0
image bump until an operator deletes the pod. The agent has no cluster
access and invented runtime state from the alert's generic annotation
text.

- issue-fix prompt: runtime claims must be attributed or framed as
  hypothesis, with read-only verification commands in the summary.
- alert-monitor prompt: when context.txt does not confirm the alert's
  symptom, say so explicitly — the mismatch is often the real clue.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

claude-bot added 1 commit 2026-07-16 08:30:32 +00:00

feat(ci-agents): give triage/fix agents verified read-only live diagnostics 90aeaf1ef8

Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:

- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
  ci-diagnostics with a read-only ClusterRole (workloads, events,
  pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
  plus a namespaced services/proxy Role for the Prometheus and
  Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
  the admin kubeconfig (no new long-lived secret), builds
  .ci-diag/kubeconfig, copies the static kubectl binary for the
  jq-only ci-claude image, smoke-tests with the restricted identity,
  and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
  Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
  binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
  hypothesis you could have settled with one read-only command but
  didn't is a defect in your output'; fix agent must test the issue's
  claims before building on them; minimal-quote rule against dumping
  logs into public issues.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

claude-bot changed title from ~~fix(ci-agents): forbid asserting unverified cluster state in triage/fix prompts~~ to feat(ci-agents): verified live diagnostics for the triage/fix agents (no more blind assessments)

2026-07-16 08:31:00 +00:00

k.renner merged commit 2505f1a46f into master

2026-07-16 08:31:38 +00:00

k.renner referenced this issue from a commit

2026-07-16 08:31:40 +00:00

Merge pull request 'feat(ci-agents): verified live diagnostics for the triage/fix agents (no more blind assessments)' (#253) from fix/agent-prompts-no-invented-state into master

claude-bot referenced this pull request

2026-07-16 08:36:11 +00:00

fix(ci-diagnostics): grant read on velero.io and postgresql.cnpg.io CRDs #254

Sign in to join this conversation.