konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

fix(node-metrics): repair dead dashboard — nonexistent 'host' label broke every panel #281

Merged
k.renner merged 1 commits from claude-bot/dash-node-metrics into master 2026-07-17 11:21:41 +00:00
Conversation 0 Commits 1 Files Changed 1
+30 -13
claude-bot commented 2026-07-16 22:43:49 +00:00
Member
Copy Link
Copy Source

Before

All five panels dead, Host dropdown empty:

before

After

Node selector working (k8s-1/2/3), all panels rendering verified data:

after

Review report

This dashboard was completely dead — all five panels blank and the Host dropdown empty.

Findings

  1. CRITICAL — template variable queries a nonexistent label. label_values(node_uname_info, host) returns nothing: no node-exporter metric in this Prometheus carries a host label. (A host label does exist server-wide, but only on blackbox-probe metrics, holding probe hostnames like alerts.renner.uno — unrelated to nodes.) The variable resolved to empty (var-host= in the URL) and every panel's host=~"$host" filter killed the query.
    ✅ FIXED — replaced with a visible $node variable (label_values(node_uname_info{job="node-exporter"}, nodename) → k8s-1/2/3) plus a hidden chained $instance variable (label_values(node_uname_info{job="node-exporter", nodename="$node"}, instance)). All panels now filter instance="$instance". This is the standard node-exporter pattern; the nodename→instance mapping was verified 1:1.

  2. MAJOR — Network I/O filter written for a Docker host, not this Cilium cluster. The exclusion lo|veth.*|br.*|docker.* misses the ~40+ per-pod Cilium interfaces (lxc*) and cilium_host/cilium_net/cilium_vxlan on every node — once the variable was fixed the panel would have drowned in dozens of meaningless series.
    ✅ FIXED — added lxc.*|cilium.*|ip6tnl.*|sit.*|teql.*|tunl.* to the exclusion (the last four are always-zero kernel pseudo-devices, verified 0 B/s). Remaining series: eth0, eth1, eth2, bond0. bond0 is kept deliberately — it is a real configured interface, currently idle.

  3. MAJOR — Disk I/O includes 12 Ceph RBD-mapped devices. node_disk_* on these nodes exposes rbd0–rbd11 (Rook-Ceph volumes mapped for pods) alongside the physical disks. That makes the panel unreadable and double-counts I/O that already hits the physical disks via Ceph.
    ✅ FIXED — excluded rbd.*|dm-.*|loop.*; the panel now shows the physical devices nvme0n1 and vda only. If per-pod RBD volume I/O is wanted, it deserves its own panel — say the word and I'll add one.

  4. MINOR — Filesystem Usage shows a meaningless rootfs bind mount. fstype!~"tmpfs|overlay" still matched rootfs /etc/extensions.yaml (0.3%, a Talos rootfs bind). The only meaningful mount on a Talos node is /var (xfs).
    ✅ FIXED — extended the exclusion to tmpfs|overlay|rootfs. The gauge now shows the single /var mount (49.2% on k8s-1 at review time, matching Prometheus: 49.16%).

  5. Not changed: panel layout, units (percent/bytes/Bps all correct), thresholds (70/85% on filesystem — sensible), rate() windows (5m ≫ scrape interval, fine), 1h default range, 30s refresh. The Memory panel's Used/Cached/Buffers stacking is a standard approximation and left as-is. The long-window scrape-residue issue (duplicate HA scrape jobs, fixed 2026-07-16) does not apply — node-exporter metrics were never scraped by the duplicate HA job, and the default window is 1h.

Verification

All against live Prometheus (port-forward, instance 10.2.40.174:9100 = k8s-1):

  • node_uname_info labels: no host label; identity is nodename + instance — root cause confirmed.
  • Fixed CPU query → 8 series (8 cores), values 20–24%.
  • Fixed memory queries → 1 series each, ~15 GiB used / 40 GiB total.
  • Fixed network query → eth0/eth1/eth2/bond0 only (was: +40 lxc*/cilium*/pseudo devices).
  • Fixed disk query → nvme0n1 + vda only (was: +12 rbd*).
  • Fixed filesystem query → single /var series, 49.16%.
  • nodename→instance chain: node_uname_info{nodename="k8s-1"} → exactly 10.2.40.174:9100.

After-render was produced via a temp preview uid (node-metrics-preview), screenshotted, and the preview deleted.

## Before All five panels dead, Host dropdown empty: ![before](https://gitea.renner.uno/attachments/fa536fc3-5ab9-4b8f-860b-a0e2ee9147ee) ## After Node selector working (k8s-1/2/3), all panels rendering verified data: ![after](https://gitea.renner.uno/attachments/974701d0-3fe3-47db-9a27-1c2fbf4b5686) ## Review report This dashboard was **completely dead** — all five panels blank and the Host dropdown empty. ### Findings 1. **CRITICAL — template variable queries a nonexistent label.** `label_values(node_uname_info, host)` returns nothing: no node-exporter metric in this Prometheus carries a `host` label. (A `host` label does exist server-wide, but only on blackbox-probe metrics, holding probe hostnames like `alerts.renner.uno` — unrelated to nodes.) The variable resolved to empty (`var-host=` in the URL) and every panel's `host=~"$host"` filter killed the query. ✅ FIXED — replaced with a visible `$node` variable (`label_values(node_uname_info{job="node-exporter"}, nodename)` → k8s-1/2/3) plus a hidden chained `$instance` variable (`label_values(node_uname_info{job="node-exporter", nodename="$node"}, instance)`). All panels now filter `instance="$instance"`. This is the standard node-exporter pattern; the nodename→instance mapping was verified 1:1. 2. **MAJOR — Network I/O filter written for a Docker host, not this Cilium cluster.** The exclusion `lo|veth.*|br.*|docker.*` misses the ~40+ per-pod Cilium interfaces (`lxc*`) and `cilium_host`/`cilium_net`/`cilium_vxlan` on every node — once the variable was fixed the panel would have drowned in dozens of meaningless series. ✅ FIXED — added `lxc.*|cilium.*|ip6tnl.*|sit.*|teql.*|tunl.*` to the exclusion (the last four are always-zero kernel pseudo-devices, verified 0 B/s). Remaining series: eth0, eth1, eth2, bond0. `bond0` is kept deliberately — it is a real configured interface, currently idle. 3. **MAJOR — Disk I/O includes 12 Ceph RBD-mapped devices.** `node_disk_*` on these nodes exposes `rbd0`–`rbd11` (Rook-Ceph volumes mapped for pods) alongside the physical disks. That makes the panel unreadable and double-counts I/O that already hits the physical disks via Ceph. ✅ FIXED — excluded `rbd.*|dm-.*|loop.*`; the panel now shows the physical devices `nvme0n1` and `vda` only. If per-pod RBD volume I/O is wanted, it deserves its own panel — say the word and I'll add one. 4. **MINOR — Filesystem Usage shows a meaningless rootfs bind mount.** `fstype!~"tmpfs|overlay"` still matched `rootfs /etc/extensions.yaml` (0.3%, a Talos rootfs bind). The only meaningful mount on a Talos node is `/var` (xfs). ✅ FIXED — extended the exclusion to `tmpfs|overlay|rootfs`. The gauge now shows the single `/var` mount (49.2% on k8s-1 at review time, matching Prometheus: 49.16%). 5. Not changed: panel layout, units (percent/bytes/Bps all correct), thresholds (70/85% on filesystem — sensible), `rate()` windows (5m ≫ scrape interval, fine), 1h default range, 30s refresh. The Memory panel's Used/Cached/Buffers stacking is a standard approximation and left as-is. The long-window scrape-residue issue (duplicate HA scrape jobs, fixed 2026-07-16) does not apply — node-exporter metrics were never scraped by the duplicate HA job, and the default window is 1h. ### Verification All against live Prometheus (port-forward, instance `10.2.40.174:9100` = k8s-1): - `node_uname_info` labels: **no `host` label**; identity is `nodename` + `instance` — root cause confirmed. - Fixed CPU query → 8 series (8 cores), values 20–24%. - Fixed memory queries → 1 series each, ~15 GiB used / 40 GiB total. - Fixed network query → eth0/eth1/eth2/bond0 only (was: +40 lxc*/cilium*/pseudo devices). - Fixed disk query → nvme0n1 + vda only (was: +12 rbd*). - Fixed filesystem query → single `/var` series, 49.16%. - nodename→instance chain: `node_uname_info{nodename="k8s-1"}` → exactly `10.2.40.174:9100`. After-render was produced via a temp preview uid (`node-metrics-preview`), screenshotted, and the preview deleted.
node-metrics-before.jpg
20 KiB
node-metrics-after.jpg
106 KiB
claude-bot added 1 commit 2026-07-16 22:43:49 +00:00
fix(node-metrics): repair dead dashboard — nonexistent 'host' label broke every panel bd8a632334 
The template variable queried label_values(node_uname_info, host), but no
node-exporter metric in this Prometheus carries a 'host' label (that label
only exists on blackbox-probe metrics). The variable resolved to empty and
all five panels rendered blank.

- Replace $host with a visible $node variable (nodename: k8s-1/2/3) plus a
  hidden chained $instance variable (nodename -> instance via node_uname_info)
- Filter Cilium pod interfaces (lxc*, cilium_*) and always-zero kernel
  pseudo-devices (ip6tnl0, sit0, teql0, tunl0) out of Network I/O
- Filter Ceph RBD-mapped volumes (rbd0..11) out of Disk I/O so the panel
  shows physical disks (nvme0n1, vda) only
- Exclude the rootfs bind mount (/etc/extensions.yaml) from Filesystem
  Usage, leaving the meaningful /var xfs mount

All queries verified against live Prometheus before commit.
k.renner merged commit 7b84daa74f into master 2026-07-17 11:21:41 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#281