feat(monitoring): expose NVMe endurance, with the privilege exception it needs #669

Merged

k.renner merged 1 commits from smartctl-exporter into master 2026-08-27 12:56:00 +00:00

Conversation 1 Commits 1 Files Changed 5

+276

claude-bot commented 2026-08-27 12:42:50 +00:00

Member

Copy Link

Copy Source

Deine Entscheidung umgesetzt: SMART für alle neun Laufwerke, inklusive der ausgemusterten Crucial, die EPHEMERAL tragen.

Die Privilegien-Ausnahme ist im Manifest dokumentiert, samt Begründung und Alternative. Sie ist so klein gehalten, wie der Kernel es zulässt: kein privileged: true, keine Host-Namespaces, kein hostPID. Root plus CAP_SYS_ADMIN und CAP_SYS_RAWIO, alles andere gedroppt, Read-only-Root, und /dev als einziger Host-Mount — nötig, weil Kubernetes einzelne Zeichengeräte ohne Device-Plugin nicht durchreichen kann. Die Admission-Policy, die privilegierte Pods verbietet, ist an woodpecker-ci gebunden; hier wird nichts umgangen.

Die Anforderung kommt vom Kernel, nicht vom Werkzeug. NVMe-Health läuft über einen Admin-ioctl auf dem Zeichengerät, gated auf CAP_SYS_ADMIN; sysfs gibt nur die Temperatur her. Ein anderer Exporter würde daran nichts ändern. Das steht so im Header — was man eines Tages neu prüfen sollte, ist ob der Kernel ein sysfs-Interface bekommen hat, nicht ob ein anderes Tool weniger verlangt.

Die verworfene Alternative ist mit festgehalten, weil sie real war: Ceph archiviert täglich vollständige smartctl-Ausgabe für die OSDs, ohne jede Ausnahme. Nur eben für drei von neun Laufwerken — und die sechs, die fehlen, sind genau die verschlissenen. Die gesunden überwachen und die kaputten dunkel lassen war den neuen Dienst nicht wert.

Zwei Details, ohne die die Metriken wertlos wären:

Der ServiceMonitor relabelt den Node auf jede Serie. Ohne das trägt jeder Messwert nur die Pod-IP des DaemonSets — die sagt nichts darüber, in welcher Maschine das Laufwerk steckt, und wechselt bei jedem Reschedule.

Keine --smartctl.device-Liste. Der Exporter scannt selbst und rescannt alle 10 Minuten; eine aufgeschriebene Geräteliste würde beim nächsten Laufwerkstausch still aufhören zu greifen. Die Gerätenamen sind hier ohnehin nicht über Nodes hinweg stabil.

Die Alarme trennen Verschleiß von Schaden. Ein Verschleißprozentsatz ist ein Budget, das ausgegeben wird — Warnung bei 80, kritisch bei 95. Erschöpfter Spare-Bereich, ein Critical-Warning-Flag oder neue Media-Errors bedeuten dagegen, dass das Laufwerk jetzt ausfällt. Media-Errors alarmieren auf den Zuwachs statt auf den Absolutwert, damit ein Laufwerk mit alten Fehlern nicht ewig alarmiert und eines, das heute anfängt, nicht untergeht.

Der Vollständigkeits-Alarm vergleicht gegen node-exporters eigenes NVMe-Inventar statt gegen eine aufgeschriebene Flottengröße — nach deinem Hinweis von vorhin. Eine Regel, die auf nichts alarmiert, sieht aus wie ein gesunder Cluster.

Dashboard-Panels kommen getrennt, sobald die Metriken laufen und ich die Label-Namen an echten Serien ablesen kann statt sie aus einer README zu raten.

Geprüft: beide Kustomizations bauen, alle drei Manifeste bestehen kubectl apply --dry-run=server. Nebenbefund: smartctl_exporter hat seit v0.14.0 (April 2025) kein Release mehr, ist aber nicht archiviert.

Closes #549

Deine Entscheidung umgesetzt: SMART für alle neun Laufwerke, inklusive der ausgemusterten Crucial, die EPHEMERAL tragen. **Die Privilegien-Ausnahme ist im Manifest dokumentiert, samt Begründung und Alternative.** Sie ist so klein gehalten, wie der Kernel es zulässt: kein `privileged: true`, keine Host-Namespaces, kein hostPID. Root plus `CAP_SYS_ADMIN` und `CAP_SYS_RAWIO`, alles andere gedroppt, Read-only-Root, und `/dev` als einziger Host-Mount — nötig, weil Kubernetes einzelne Zeichengeräte ohne Device-Plugin nicht durchreichen kann. Die Admission-Policy, die privilegierte Pods verbietet, ist an `woodpecker-ci` gebunden; hier wird nichts umgangen. **Die Anforderung kommt vom Kernel, nicht vom Werkzeug.** NVMe-Health läuft über einen Admin-ioctl auf dem Zeichengerät, gated auf `CAP_SYS_ADMIN`; sysfs gibt nur die Temperatur her. Ein anderer Exporter würde daran nichts ändern. Das steht so im Header — was man eines Tages neu prüfen sollte, ist ob der Kernel ein sysfs-Interface bekommen hat, nicht ob ein anderes Tool weniger verlangt. **Die verworfene Alternative ist mit festgehalten**, weil sie real war: Ceph archiviert täglich vollständige smartctl-Ausgabe für die OSDs, ohne jede Ausnahme. Nur eben für drei von neun Laufwerken — und die sechs, die fehlen, sind genau die verschlissenen. Die gesunden überwachen und die kaputten dunkel lassen war den neuen Dienst nicht wert. **Zwei Details, ohne die die Metriken wertlos wären:** Der ServiceMonitor relabelt den Node auf jede Serie. Ohne das trägt jeder Messwert nur die Pod-IP des DaemonSets — die sagt nichts darüber, in welcher Maschine das Laufwerk steckt, und wechselt bei jedem Reschedule. Keine `--smartctl.device`-Liste. Der Exporter scannt selbst und rescannt alle 10 Minuten; eine aufgeschriebene Geräteliste würde beim nächsten Laufwerkstausch still aufhören zu greifen. Die Gerätenamen sind hier ohnehin nicht über Nodes hinweg stabil. **Die Alarme trennen Verschleiß von Schaden.** Ein Verschleißprozentsatz ist ein Budget, das ausgegeben wird — Warnung bei 80, kritisch bei 95. Erschöpfter Spare-Bereich, ein Critical-Warning-Flag oder neue Media-Errors bedeuten dagegen, dass das Laufwerk *jetzt* ausfällt. Media-Errors alarmieren auf den Zuwachs statt auf den Absolutwert, damit ein Laufwerk mit alten Fehlern nicht ewig alarmiert und eines, das heute anfängt, nicht untergeht. Der Vollständigkeits-Alarm vergleicht gegen node-exporters eigenes NVMe-Inventar statt gegen eine aufgeschriebene Flottengröße — nach deinem Hinweis von vorhin. Eine Regel, die auf nichts alarmiert, sieht aus wie ein gesunder Cluster. **Dashboard-Panels kommen getrennt**, sobald die Metriken laufen und ich die Label-Namen an echten Serien ablesen kann statt sie aus einer README zu raten. Geprüft: beide Kustomizations bauen, alle drei Manifeste bestehen `kubectl apply --dry-run=server`. Nebenbefund: smartctl_exporter hat seit v0.14.0 (April 2025) kein Release mehr, ist aber nicht archiviert. Closes #549

claude-bot added 1 commit 2026-08-27 12:42:51 +00:00

feat(monitoring): expose NVMe endurance, with the privilege exception it needs ... e8b217e8a1

Drive wear has been invisible in every signal this cluster has. `ceph -s` reads
active+clean at any wear level, no Ceph health check fires until a device
actually fails, and the block-layer metrics show throughput and latency but
never endurance. That blind spot is how a set of OSD drives reached 81-87%
before anyone looked (#549, #434).

This closes it for every drive, and it costs a deliberate exception to the
no-privileged-containers rule. The owner made that call explicitly after being
shown the alternative.

The alternative was real and is worth recording. Ceph's devicehealth module
already archives complete smartctl output daily — percentage_used,
power_on_hours, media_errors, critical_warning, available_spare — collected
without any elevated privileges, because the OSD pods hold the device access
already. Exporting that needs no exception at all. It also covers OSDs and
nothing else: three of nine drives here, and the six it misses include the ones
retired from Ceph for wear that now carry EPHEMERAL. Monitoring the healthy
drives while leaving the worn ones dark was not worth a new service.

The requirement comes from the kernel, not from the tool. NVMe health is read
through an admin ioctl on the character device, which is gated on CAP_SYS_ADMIN;
sysfs publishes the controller temperature and nothing more. Swapping
smartctl_exporter for another exporter would not change that, so the header says
so — the thing to re-check one day is whether the kernel grew a sysfs interface,
not whether some other tool asks for less.

The exception is kept to the smallest form that works: no `privileged: true`, no
host namespaces, no hostPID. Root plus CAP_SYS_ADMIN and CAP_SYS_RAWIO, all
other capabilities dropped, a read-only root filesystem, and /dev as the only
host mount — required because Kubernetes cannot hand individual character
devices to a pod without a device plugin. The admission policy that forbids
privileged pods is bound to the woodpecker-ci namespace only, so nothing here
circumvents it; monitoring already enforces pod-security `privileged`.

Two details that would otherwise have made the metrics useless:

The ServiceMonitor relabels the pod's node onto every series. Without it each
sample carries only the DaemonSet pod's IP, which says nothing about which
machine the drive is in and changes on every reschedule — an unattributable wear
figure is most of the value gone.

No --smartctl.device list. The exporter scans and rescans on its own, and a
written-down device list would quietly stop covering a drive the next time one
is replaced. Device names are not stable across these machines in the first
place.

The alert rules deliberately separate wear from damage: a wear percentage is a
budget being spent and warns at 80, while spare exhaustion, a critical-warning
flag or new media errors mean the drive is failing now. Media errors alert on
the increase rather than the total, so a drive that recorded errors long ago
does not alert forever while one that starts failing today does.

The completeness check compares the number of reporting drives against
node-exporter's own NVMe inventory rather than a written-down fleet size, so
adding or removing a drive moves both sides together. A rule that alerts on
nothing looks exactly like a healthy cluster, which is the failure mode worth
guarding.

Dashboard panels follow separately, once the metrics are live and the label
names can be read off the real series rather than guessed from a README.

Closes #549

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot requested changes 2026-08-27 12:47:10 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit e8b217e8a1aee78708b9007e17f2f25b180df6ef.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

Adds a smartctl-exporter DaemonSet + headless Service to monitoring, plus a ServiceMonitor and a PrometheusRule (disk-wear.rules) in monitoring-rules, to expose and alert on NVMe wear/health across all nine drives. The elevated-privilege rationale (root + CAP_SYS_ADMIN/CAP_SYS_RAWIO, no privileged: true, no host namespaces) is well-documented and consistent with existing precedent in this repo (monitoring namespace already runs pod-security.kubernetes.io/enforce: privileged, and the woodpecker-ci admission policy the PR body cites is in fact scoped only to that namespace — confirmed in kubernetes/infrastructure/admission-policies/woodpecker-ci-restrict.yaml). One real bug: the alert annotations don't use the node-identifying label the ServiceMonitor was specifically added to produce.

🔴 Blocking

• kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:62,76,91,106,121 — every alert annotation interpolates {{ $labels.instance }}, but the ServiceMonitor (kubernetes/apps/monitoring-rules/smartctl-exporter-servicemonitor.yaml:28-29) only relabels __meta_kubernetes_pod_node_name onto a new node label — it does not touch instance. Since kube-prometheus-stack's ServiceMonitor scraping leaves instance as the target's pod IP:port (no other relabeling in this repo overrides it — checked helmrelease-prometheus.yaml and every other *-servicemonitor.yaml in monitoring-rules/), every fired alert will show something like NVMe /dev/nvme0 on 10.244.x.x:9633 is at 82% rated wear instead of the actual machine. This is exactly the failure mode the PR body calls out as the reason the relabeling exists ("Ohne das trägt jeder Messwert nur die Pod-IP des DaemonSets... und wechselt bei jedem Reschedule") — the relabeling was added but the alert text never adopted it, so the stated purpose isn't actually achieved. Fix: use {{ $labels.node }} in the five annotation strings (or additionally relabel instance itself).

🟡 Suggestions

• kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:134 (SmartctlExporterIncomplete) — count(smartctl_device_percentage_used) < count(node_nvme_info) assumes a 1:1 series relationship between the two metrics. This is very likely correct (node_exporter's nvme collector and smartctl_exporter's NVMe collector both key off the controller device, not per-namespace), and the existing disk-health.json dashboard already relies on node_nvme_info the same way, but it's worth a quick sanity check against real series once the exporter is running, since a permanent mismatch here would make this alert permanently red (or permanently silent) rather than a useful completeness check.
• kubernetes/apps/monitoring/smartctl-exporter.yaml:266-275 — CAP_SYS_RAWIO is added alongside CAP_SYS_ADMIN. The PR's own rationale is entirely about the NVMe admin ioctl, which is gated on CAP_SYS_ADMIN alone; CAP_SYS_RAWIO is typically needed for SCSI/ATA SG_IO passthrough on non-NVMe drives. If this fleet is NVMe-only (as the PR title/body imply), it may be droppable — worth a one-time check of whether smartctl actually needs it here, given the stated goal of keeping the exception "as small as the kernel allows."
• kubernetes/apps/monitoring/smartctl-exporter.yaml:292-297 — only a livenessProbe is defined, no readinessProbe. Without one, the pod is added to the Service's Endpoints (and thus scraped) as soon as the container starts, before /metrics is guaranteed to be serving; harmless here (Prometheus scrape just fails/retries) but a readinessProbe mirroring the liveness check would be more consistent with typical exporter patterns.

🟢 Nits

• kubernetes/apps/monitoring/smartctl-exporter.yaml — no automountServiceAccountToken: false. The pod doesn't call the API server, so mounting the default SA token is unnecessary attack surface, though minor given the pod already runs as root with CAP_SYS_ADMIN.

Tests

No automated tests exist for this class of change in this repo (Kustomize/PrometheusRule manifests), consistent with the rest of monitoring-rules/. The PR body states both Kustomizations build and all three manifests pass kubectl apply --dry-run=server, which is the right validation bar here. The one thing that can't be dry-run-validated is exactly the bug found above — annotation label references aren't checked by admission, only by looking at fired alerts — so this class of mistake is expected to slip past that validation and is worth a manual check against a real alert once the exporter is live.

Recommendation

Request changes — 1 blocking issue.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `e8b217e8a1aee78708b9007e17f2f25b180df6ef`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a `smartctl-exporter` DaemonSet + headless Service to `monitoring`, plus a `ServiceMonitor` and a `PrometheusRule` (`disk-wear.rules`) in `monitoring-rules`, to expose and alert on NVMe wear/health across all nine drives. The elevated-privilege rationale (root + `CAP_SYS_ADMIN`/`CAP_SYS_RAWIO`, no `privileged: true`, no host namespaces) is well-documented and consistent with existing precedent in this repo (`monitoring` namespace already runs `pod-security.kubernetes.io/enforce: privileged`, and the `woodpecker-ci` admission policy the PR body cites is in fact scoped only to that namespace — confirmed in `kubernetes/infrastructure/admission-policies/woodpecker-ci-restrict.yaml`). One real bug: the alert annotations don't use the node-identifying label the ServiceMonitor was specifically added to produce. ## 🔴 Blocking - `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:62,76,91,106,121` — every alert annotation interpolates `{{ $labels.instance }}`, but the `ServiceMonitor` (`kubernetes/apps/monitoring-rules/smartctl-exporter-servicemonitor.yaml:28-29`) only relabels `__meta_kubernetes_pod_node_name` onto a new `node` label — it does not touch `instance`. Since kube-prometheus-stack's ServiceMonitor scraping leaves `instance` as the target's pod IP:port (no other relabeling in this repo overrides it — checked `helmrelease-prometheus.yaml` and every other `*-servicemonitor.yaml` in `monitoring-rules/`), every fired alert will show something like `NVMe /dev/nvme0 on 10.244.x.x:9633 is at 82% rated wear` instead of the actual machine. This is exactly the failure mode the PR body calls out as the reason the relabeling exists ("Ohne das trägt jeder Messwert nur die Pod-IP des DaemonSets... und wechselt bei jedem Reschedule") — the relabeling was added but the alert text never adopted it, so the stated purpose isn't actually achieved. Fix: use `{{ $labels.node }}` in the five annotation strings (or additionally relabel `instance` itself). ## 🟡 Suggestions - `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:134` (`SmartctlExporterIncomplete`) — `count(smartctl_device_percentage_used) < count(node_nvme_info)` assumes a 1:1 series relationship between the two metrics. This is very likely correct (node_exporter's `nvme` collector and smartctl_exporter's NVMe collector both key off the controller device, not per-namespace), and the existing `disk-health.json` dashboard already relies on `node_nvme_info` the same way, but it's worth a quick sanity check against real series once the exporter is running, since a permanent mismatch here would make this alert permanently red (or permanently silent) rather than a useful completeness check. - `kubernetes/apps/monitoring/smartctl-exporter.yaml:266-275` — `CAP_SYS_RAWIO` is added alongside `CAP_SYS_ADMIN`. The PR's own rationale is entirely about the NVMe admin ioctl, which is gated on `CAP_SYS_ADMIN` alone; `CAP_SYS_RAWIO` is typically needed for SCSI/ATA `SG_IO` passthrough on non-NVMe drives. If this fleet is NVMe-only (as the PR title/body imply), it may be droppable — worth a one-time check of whether smartctl actually needs it here, given the stated goal of keeping the exception "as small as the kernel allows." - `kubernetes/apps/monitoring/smartctl-exporter.yaml:292-297` — only a `livenessProbe` is defined, no `readinessProbe`. Without one, the pod is added to the Service's Endpoints (and thus scraped) as soon as the container starts, before `/metrics` is guaranteed to be serving; harmless here (Prometheus scrape just fails/retries) but a `readinessProbe` mirroring the liveness check would be more consistent with typical exporter patterns. ## 🟢 Nits - `kubernetes/apps/monitoring/smartctl-exporter.yaml` — no `automountServiceAccountToken: false`. The pod doesn't call the API server, so mounting the default SA token is unnecessary attack surface, though minor given the pod already runs as root with `CAP_SYS_ADMIN`. ## Tests No automated tests exist for this class of change in this repo (Kustomize/PrometheusRule manifests), consistent with the rest of `monitoring-rules/`. The PR body states both Kustomizations build and all three manifests pass `kubectl apply --dry-run=server`, which is the right validation bar here. The one thing that can't be dry-run-validated is exactly the bug found above — annotation label references aren't checked by admission, only by looking at fired alerts — so this class of mistake is expected to slip past that validation and is worth a manual check against a real alert once the exporter is live. ## Recommendation Request changes — 1 blocking issue. <!-- pr-review:sha=e8b217e8a1aee78708b9007e17f2f25b180df6ef -->

k.renner merged commit 04df5da041 into master 2026-08-27 12:56:00 +00:00

k.renner referenced this issue from a commit 2026-08-27 12:56:00 +00:00

Merge pull request 'feat(monitoring): expose NVMe endurance, with the privilege exception it needs' (#669) from smartctl-exporter into master

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#669