fix: [alert] CephHealthError (rook-ceph) (#525) #526

2026-08-19T16:46:26Z

issue-fix-bot commented

2026-08-19 16:46:26 +00:00

What this PR does

CephHealthError has been firing since 15:53 UTC because Ceph really is in HEALTH_ERR — but nothing is at risk. The v19.2.5 → v19.2.6 patch upgrade (694389b) shipped six new AUTH_INSECURE_* cephx health checks, two of them ERR-severity, which flag the legacy aes key type that every keyring in this cluster already used. A reporting change, not a regression.

The real damage is not the noise, it's the blindness: ceph_health_status == 2 is now permanently saturated, so a genuine PG_AVAILABILITY or OSD_FULL would be indistinguishable from the auth finding and would never page. This PR narrows CephHealthError to ERR-severity checks that are not the auth family, restoring the cluster's ability to page for real availability loss, and adds CephAuthInsecureKeys (warning) to carry the auth posture so it is tracked rather than dropped. It does not attempt the cephx key rotation — that is live-cluster state, not manifest state (see Notes).

Changes

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml — CephHealthError now fires on ceph_health_status == 2 unless every active ERR check is an AUTH_INSECURE_* one; adds CephAuthInsecureKeys (severity warning, for: 1h) covering the excluded condition. 42 of the 44 changed lines are comments/annotations, matching this file's existing style; the logic is ~14 lines.

Verification

All claims below were observed in command output this run, via ./.ci-diag/kubectl get --raw against the Prometheus proxy.

The alert is real and still active. ceph_health_status = 2, ALERTS{alertname="CephHealthError"} = 1, alertstate="firing".

The ERR is auth-only. ceph_health_detail{namespace="rook-ceph"} > 0 returns exactly six series: ERR on AUTH_INSECURE_SERVICE_KEY_TYPE and AUTH_INSECURE_SERVICE_TICKETS, WARN on AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED, AUTH_INSECURE_KEYS_CREATABLE, AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE. No non-auth check is active.

Storage is genuinely healthy. Across all 12 pools ceph_pg_active == ceph_pg_clean == ceph_pg_total, and ceph_pg_degraded/ceph_pg_inconsistent/ceph_pg_undersized are 0 for every pool. ceph_osd_up and ceph_osd_in = 1 for osd.0/1/2. All 8 daemons report ceph version 19.2.6 ... squid (stable), and kubectl -n rook-ceph get pods shows every Ceph daemon Running/Ready with 0 restarts.

The new expression was tested live before committing, all four paths:

Case	Result
Current state (both ERR checks are auth)	returns empty → alert clears
Simulated non-auth ERR (allowlist narrowed so counts differ, 2 vs 1)	returns the series → alert fires
`ceph_health_detail` absent entirely (queried a nonexistent metric name)	returns the series → fails open, still pages
New `CephAuthInsecureKeys` expr	returns `6` → fires, and `{{ $value }}` renders the check count

One finding that changed the implementation: count(ceph_health_detail) = 31 while count(ceph_health_detail > 0) = 6 — the mgr exports all known checks, inactive ones at 0. Without the > 0 filters the two counts would have compared constants and the rule would have been silently broken. The same query also confirmed the metric covers the real data-risk checks (PG_AVAILABILITY, PG_DEGRADED, OSD_DOWN, OSD_FULL, MDS_ALL_DOWN, …), which is what makes the detail-based approach safe.

After Flux reconciles: CephHealthError should resolve within ~5m while ceph_health_status stays 2; CephAuthInsecureKeys should fire ~1h later. Re-run the four queries above to confirm. Reviewer sanity check — this should return empty now and non-empty during any real outage:
./.ci-diag/kubectl get --raw "/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=ceph_health_detail%7Bseverity%3D%22HEALTH_ERR%22%2Cname%21~%22AUTH_INSECURE_.%2A%22%7D%3E0"

Notes

This narrows an alert, which needs justifying. It is not silencing: the excluded condition moves to a dedicated warning alert, and the ERR-severity threshold is unchanged for every other check. The issue established that the rule's stated premise ("HEALTH_ERR means data availability or redundancy is at risk") became false in 19.2.6, and I confirmed that independently — the premise is now restored by construction rather than by comment.
The key rotation is deliberately not attempted. It is live-cluster state, touching client.admin, both CSI provisioner/node key pairs and the RGW key. The CSI keys are consumed by Rook-managed secrets, so careless rotation breaks RBD/CephFS provisioning cluster-wide. A human should plan it against the 19.2.6 release notes, driven through Rook or the toolbox; any durable config belongs in spec.cephConfig in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml. CephAuthInsecureKeys will keep this visible until then.
The issue's diagnosis held up in full — unusually, every runtime claim I re-checked was accurate. Its "hypothesis, not verified" caveat about the remediation mechanism remains unverified here too: I have no Ceph admin socket access, so the ceph config/ceph auth commands it suggests still need running by a human. Nothing in this PR depends on their outcome.
Stale-evidence confirmation: the Startup probe failed warnings on rook-ceph-osd-0/1/2 and the waiting for ceph monitors upgrade to finish reconcile errors were indeed the rolling upgrade, as the issue suspected — all three OSDs are now Running with 0 restarts.
Deliberate non-change: CephHealthWarn (ceph_health_status == 1, 6h) is left alone. Once the two ERR auth checks are remediated the four WARN ones would drop the cluster to HEALTH_WARN and trip it, duplicating CephAuthInsecureKeys. That is contingent on future remediation, is warning-severity only, and fixing it now would mean guessing at the post-rotation state — worth revisiting as part of the rotation work.
Out of scope, flagged not actioned: the issue suggests reviewing kubernetes/infrastructure/image-automation/ceph.yaml, which auto-merges any 19.x patch with no human gate — that is how a health-check behaviour change landed unattended. That is a policy decision for the owner, not an alert fix, so I made no change there.
No image tags were touched and no pinned components bumped.

Closes #525

## What this PR does `CephHealthError` has been firing since 15:53 UTC because Ceph really is in HEALTH_ERR — but nothing is at risk. The v19.2.5 → v19.2.6 patch upgrade (`694389b`) shipped six new `AUTH_INSECURE_*` cephx health checks, two of them ERR-severity, which flag the legacy `aes` key type that every keyring in this cluster already used. A reporting change, not a regression. The real damage is not the noise, it's the **blindness**: `ceph_health_status == 2` is now permanently saturated, so a genuine `PG_AVAILABILITY` or `OSD_FULL` would be indistinguishable from the auth finding and would never page. This PR narrows `CephHealthError` to ERR-severity checks that are *not* the auth family, restoring the cluster's ability to page for real availability loss, and adds `CephAuthInsecureKeys` (warning) to carry the auth posture so it is tracked rather than dropped. It does **not** attempt the cephx key rotation — that is live-cluster state, not manifest state (see Notes). ## Changes - `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml` — `CephHealthError` now fires on `ceph_health_status == 2` *unless every active ERR check is an `AUTH_INSECURE_*` one*; adds `CephAuthInsecureKeys` (severity `warning`, `for: 1h`) covering the excluded condition. 42 of the 44 changed lines are comments/annotations, matching this file's existing style; the logic is ~14 lines. ## Verification All claims below were observed in command output this run, via `./.ci-diag/kubectl get --raw` against the Prometheus proxy. **The alert is real and still active.** `ceph_health_status` = `2`, `ALERTS{alertname="CephHealthError"}` = `1`, `alertstate="firing"`. **The ERR is auth-only.** `ceph_health_detail{namespace="rook-ceph"} > 0` returns exactly six series: ERR on `AUTH_INSECURE_SERVICE_KEY_TYPE` and `AUTH_INSECURE_SERVICE_TICKETS`, WARN on `AUTH_INSECURE_CLIENT_KEY_TYPE`, `AUTH_INSECURE_KEYS_ALLOWED`, `AUTH_INSECURE_KEYS_CREATABLE`, `AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE`. No non-auth check is active. **Storage is genuinely healthy.** Across all 12 pools `ceph_pg_active` == `ceph_pg_clean` == `ceph_pg_total`, and `ceph_pg_degraded`/`ceph_pg_inconsistent`/`ceph_pg_undersized` are `0` for every pool. `ceph_osd_up` and `ceph_osd_in` = `1` for osd.0/1/2. All 8 daemons report `ceph version 19.2.6 ... squid (stable)`, and `kubectl -n rook-ceph get pods` shows every Ceph daemon Running/Ready with 0 restarts. **The new expression was tested live before committing**, all four paths: | Case | Result | |---|---| | Current state (both ERR checks are auth) | returns empty → **alert clears** | | Simulated non-auth ERR (allowlist narrowed so counts differ, 2 vs 1) | returns the series → **alert fires** | | `ceph_health_detail` absent entirely (queried a nonexistent metric name) | returns the series → **fails open, still pages** | | New `CephAuthInsecureKeys` expr | returns `6` → fires, and `{{ $value }}` renders the check count | **One finding that changed the implementation:** `count(ceph_health_detail)` = `31` while `count(ceph_health_detail > 0)` = `6` — the mgr exports *all* known checks, inactive ones at `0`. Without the `> 0` filters the two counts would have compared constants and the rule would have been silently broken. The same query also confirmed the metric covers the real data-risk checks (`PG_AVAILABILITY`, `PG_DEGRADED`, `OSD_DOWN`, `OSD_FULL`, `MDS_ALL_DOWN`, …), which is what makes the detail-based approach safe. **After Flux reconciles:** `CephHealthError` should resolve within ~5m while `ceph_health_status` stays `2`; `CephAuthInsecureKeys` should fire ~1h later. Re-run the four queries above to confirm. Reviewer sanity check — this should return empty now and non-empty during any real outage: `./.ci-diag/kubectl get --raw "/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=ceph_health_detail%7Bseverity%3D%22HEALTH_ERR%22%2Cname%21~%22AUTH_INSECURE_.%2A%22%7D%3E0"` ## Notes - **This narrows an alert, which needs justifying.** It is not silencing: the excluded condition moves to a dedicated warning alert, and the ERR-severity threshold is unchanged for every other check. The issue established that the rule's stated premise ("HEALTH_ERR means data availability or redundancy is at risk") became false in 19.2.6, and I confirmed that independently — the premise is now restored by construction rather than by comment. - **The key rotation is deliberately not attempted.** It is live-cluster state, touching `client.admin`, both CSI provisioner/node key pairs and the RGW key. The CSI keys are consumed by Rook-managed secrets, so careless rotation breaks RBD/CephFS provisioning cluster-wide. A human should plan it against the 19.2.6 release notes, driven through Rook or the toolbox; any durable config belongs in `spec.cephConfig` in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml`. `CephAuthInsecureKeys` will keep this visible until then. - **The issue's diagnosis held up in full** — unusually, every runtime claim I re-checked was accurate. Its "hypothesis, not verified" caveat about the remediation mechanism remains unverified here too: I have no Ceph admin socket access, so the `ceph config`/`ceph auth` commands it suggests still need running by a human. Nothing in this PR depends on their outcome. - **Stale-evidence confirmation:** the `Startup probe failed` warnings on `rook-ceph-osd-0/1/2` and the `waiting for ceph monitors upgrade to finish` reconcile errors were indeed the rolling upgrade, as the issue suspected — all three OSDs are now Running with 0 restarts. - **Deliberate non-change:** `CephHealthWarn` (`ceph_health_status == 1`, 6h) is left alone. Once the two ERR auth checks are remediated the four WARN ones would drop the cluster to HEALTH_WARN and trip it, duplicating `CephAuthInsecureKeys`. That is contingent on future remediation, is warning-severity only, and fixing it now would mean guessing at the post-rotation state — worth revisiting as part of the rotation work. - **Out of scope, flagged not actioned:** the issue suggests reviewing `kubernetes/infrastructure/image-automation/ceph.yaml`, which auto-merges any `19.x` patch with no human gate — that is how a health-check behaviour change landed unattended. That is a policy decision for the owner, not an alert fix, so I made no change there. - No image tags were touched and no pinned components bumped. Closes #525

issue-fix-bot added 1 commit 2026-08-19 16:46:27 +00:00

fix: [alert] CephHealthError (rook-ceph) (#525 )

ci/woodpecker/pr/ci-manifests-test Pipeline is pending

Details

ci/woodpecker/pr/ci-scripts-test Pipeline is pending

Details

ci/woodpecker/pr/pr-review Pipeline is pending

Details

93ba61bf70

issue-fix-bot referenced this pull request

2026-08-19 16:46:29 +00:00

[alert] CephHealthError (rook-ceph) #525

k.renner closed this pull request

2026-08-20 04:39:47 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#526