konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

fix(node-overview): repair 4 dead panels, exclude rbd/lxc devices, nodename legends #289

Merged
k.renner merged 1 commits from claude-bot/dash-node-overview into master 2026-07-17 13:17:02 +00:00
Conversation 0 Commits 1 Files Changed 1
+17 -17
claude-bot commented 2026-07-16 23:10:55 +00:00
Member
Copy Link
Copy Source

Before

4 of 12 panels dead (Nodes Online, Nodes Down, Avg Disk Usage, Disk Usage by Node), IP:port legends, inflated I/O:

before

After

All 12 panels live, k8s-1/2/3 legends, physical-device I/O only (page zoomed out so the bottom panel is visible):

after

Review report

4 of 12 panels were completely dead — same class of defect as the node-metrics sibling (wrong labels for this cluster), just without the template-variable indirection (this dashboard has no variables, so the empty-dropdown root cause does not apply here).

Critical

  1. "Nodes Online" showed No data — query used up{job="node"}, but the scrape job here is node-exporter (verified: count by (job)(up) lists node-exporter, no node).
    ✅ FIXED — job="node-exporter"; thresholds changed to red-base/green-at-3 so a down node turns the stat red instead of staying green.
  2. "Nodes Down" showed No data (in red) even though 0 nodes are down — same wrong job label, plus count(... == 0) returns an empty vector when nothing matches, and the base threshold was red.
    ✅ FIXED — count(up{job="node-exporter"} == 0) OR on() vector(0) renders an honest 0; thresholds green-at-0 / red-at-1.
  3. "Avg Disk Usage" showed No data — filtered mountpoint="/", which does not exist in node_filesystem metrics on Talos. The only real filesystem is /var (xfs); everything else is tmpfs/rootfs bind mounts.
    ✅ FIXED — mountpoint="/var". Live value: 58.9 % fleet average.
  4. "Disk Usage by Node" (bottom bargauge) was empty — same mountpoint="/" bug.
    ✅ FIXED — /var per node: k8s-1 49.2 %, k8s-2 62.3 %, k8s-3 65.3 % (matches Prometheus exactly).

Major

  1. Disk I/O counted the ~12 Rook-Ceph rbd* mapped volumes per node on top of the physical devices, i.e. Ceph client I/O was added to the OSD I/O it becomes. rbd reads alone were ~438 kB/s vs 8–129 kB/s physical at verification time.
    ✅ FIXED — device=~"nvme.*|vd.*" (physical devices here are nvme0n1 + vda).
  2. Network I/O filter was written for a Docker host (!~"lo|veth.*|docker.*|br-.*") and therefore let ~40 Cilium lxc* per-pod interfaces plus kernel pseudo-devices (tunl0, sit0, teql0) through — pod traffic was double-counted into the per-node sum.
    ✅ FIXED — device=~"eth.*". Real traffic is on eth0/1/2; bond0 exists but is down with 0 slaves (verified node_bonding_slaves = 0), so it is deliberately excluded — if bonding is ever activated, the member eth counters still carry the traffic.

Minor / UX

  1. Legends showed 10.2.40.x:9100 instead of node names.
    ✅ FIXED — all per-node queries join node_uname_info via * on(instance) group_left(nodename) ...; legends now say k8s-1/2/3. Both sides of the join are wrapped in max by (...): over a 30d window there are 4 duplicate series per instance (node-exporter pod restarts + a Talos kernel upgrade changed the release label), which would otherwise rainbow-duplicate long-window views and can make group_left error out. Verified the guarded form collapses to exactly 1 series per node.
  2. The three "Avg …" stat panels reduced with mean, i.e. they showed a time-average over the selected window rather than the current value (a 1-week window would show "avg CPU last week", which is not what a fleet-health stat should say).
    ✅ FIXED — reduce lastNotNull; the sparkline still shows the window history.

Not changed (deliberately)

  • ⚠️ NOT FIXED — per-node write I/O still looks similar to before (~3 MB/s): that is real, because Ceph replication lands on the local nvme OSDs. The fix removes the client-side rbd double count, not the OSD write load.
  • §7 duplicate-HA-scrape residue: N/A for current data (node-exporter was never scraped by the duplicate HA job), but the max by guards in item 7 defensively cover the historical pod-restart duplicates that do exist.

Verification

All against live Prometheus (kubectl port-forward → /api/v1/query), before applying:

  • count by (job)(up) → job is node-exporter (3 targets), no job="node" anywhere.
  • node_uname_info → 1:1 instance→nodename mapping (10.2.40.174=k8s-1, .136=k8s-2, .168=k8s-3).
  • count by (mountpoint, fstype)(node_filesystem_size_bytes) → only /var (xfs) is a real filesystem; no / mountpoint exists.
  • Network devices: eth0/1/2 carry traffic; bond0 = 0 B/s, node_bonding_slaves{master="bond0"} = 0 on all nodes; ~40 lxc* + pseudo devices excluded.
  • Disk devices: nvme0n1, vda, rbd0–rbd11 per node; rbd excluded.
  • Every final panel expression executed against Prometheus and returned exactly 3 series (one per node) with plausible values (CPU 20–25 %, mem 27–33 %, /var 49–65 %).
  • count by (instance)(last_over_time(node_uname_info[30d])) = 4 → historical duplicate series exist; the max by join guards collapse them (verified).
  • AFTER screenshot rendered from a temp preview uid (node-overview-preview), then deleted.
## Before 4 of 12 panels dead (Nodes Online, Nodes Down, Avg Disk Usage, Disk Usage by Node), IP:port legends, inflated I/O: ![before](https://gitea.renner.uno/attachments/44c7000f-e181-4f73-9e03-f2d88bdce0de) ## After All 12 panels live, k8s-1/2/3 legends, physical-device I/O only (page zoomed out so the bottom panel is visible): ![after](https://gitea.renner.uno/attachments/a2060910-8372-4dbb-8cbf-18aa0ecb6892) ## Review report **4 of 12 panels were completely dead** — same class of defect as the node-metrics sibling (wrong labels for this cluster), just without the template-variable indirection (this dashboard has no variables, so the empty-dropdown root cause does not apply here). ### Critical 1. **"Nodes Online" showed No data** — query used `up{job="node"}`, but the scrape job here is `node-exporter` (verified: `count by (job)(up)` lists `node-exporter`, no `node`). ✅ FIXED — `job="node-exporter"`; thresholds changed to red-base/green-at-3 so a down node turns the stat red instead of staying green. 2. **"Nodes Down" showed No data (in red) even though 0 nodes are down** — same wrong job label, plus `count(... == 0)` returns an empty vector when nothing matches, and the base threshold was red. ✅ FIXED — `count(up{job="node-exporter"} == 0) OR on() vector(0)` renders an honest `0`; thresholds green-at-0 / red-at-1. 3. **"Avg Disk Usage" showed No data** — filtered `mountpoint="/"`, which does not exist in node_filesystem metrics on Talos. The only real filesystem is `/var` (xfs); everything else is tmpfs/rootfs bind mounts. ✅ FIXED — `mountpoint="/var"`. Live value: 58.9 % fleet average. 4. **"Disk Usage by Node" (bottom bargauge) was empty** — same `mountpoint="/"` bug. ✅ FIXED — `/var` per node: k8s-1 49.2 %, k8s-2 62.3 %, k8s-3 65.3 % (matches Prometheus exactly). ### Major 5. **Disk I/O counted the ~12 Rook-Ceph `rbd*` mapped volumes per node** on top of the physical devices, i.e. Ceph client I/O was added to the OSD I/O it becomes. rbd reads alone were ~438 kB/s vs 8–129 kB/s physical at verification time. ✅ FIXED — `device=~"nvme.*|vd.*"` (physical devices here are nvme0n1 + vda). 6. **Network I/O filter was written for a Docker host** (`!~"lo|veth.*|docker.*|br-.*"`) and therefore let ~40 Cilium `lxc*` per-pod interfaces plus kernel pseudo-devices (tunl0, sit0, teql0) through — pod traffic was double-counted into the per-node sum. ✅ FIXED — `device=~"eth.*"`. Real traffic is on eth0/1/2; `bond0` exists but is down with 0 slaves (verified `node_bonding_slaves` = 0), so it is deliberately excluded — if bonding is ever activated, the member eth counters still carry the traffic. ### Minor / UX 7. **Legends showed `10.2.40.x:9100`** instead of node names. ✅ FIXED — all per-node queries join `node_uname_info` via `* on(instance) group_left(nodename) ...`; legends now say k8s-1/2/3. Both sides of the join are wrapped in `max by (...)`: over a 30d window there are 4 duplicate series per instance (node-exporter pod restarts + a Talos kernel upgrade changed the `release` label), which would otherwise rainbow-duplicate long-window views and can make `group_left` error out. Verified the guarded form collapses to exactly 1 series per node. 8. **The three "Avg …" stat panels reduced with `mean`**, i.e. they showed a time-average over the selected window rather than the current value (a 1-week window would show "avg CPU last week", which is not what a fleet-health stat should say). ✅ FIXED — reduce `lastNotNull`; the sparkline still shows the window history. ### Not changed (deliberately) - ⚠️ NOT FIXED — per-node write I/O still looks similar to before (~3 MB/s): that is real, because Ceph replication lands on the local nvme OSDs. The fix removes the *client-side* rbd double count, not the OSD write load. - §7 duplicate-HA-scrape residue: N/A for current data (node-exporter was never scraped by the duplicate HA job), but the `max by` guards in item 7 defensively cover the historical pod-restart duplicates that do exist. ## Verification All against live Prometheus (`kubectl port-forward` → `/api/v1/query`), before applying: - `count by (job)(up)` → job is `node-exporter` (3 targets), no `job="node"` anywhere. - `node_uname_info` → 1:1 instance→nodename mapping (10.2.40.174=k8s-1, .136=k8s-2, .168=k8s-3). - `count by (mountpoint, fstype)(node_filesystem_size_bytes)` → only `/var` (xfs) is a real filesystem; **no `/` mountpoint exists**. - Network devices: eth0/1/2 carry traffic; bond0 = 0 B/s, `node_bonding_slaves{master="bond0"}` = 0 on all nodes; ~40 `lxc*` + pseudo devices excluded. - Disk devices: nvme0n1, vda, rbd0–rbd11 per node; rbd excluded. - Every final panel expression executed against Prometheus and returned exactly 3 series (one per node) with plausible values (CPU 20–25 %, mem 27–33 %, /var 49–65 %). - `count by (instance)(last_over_time(node_uname_info[30d]))` = 4 → historical duplicate series exist; the `max by` join guards collapse them (verified). - AFTER screenshot rendered from a temp preview uid (`node-overview-preview`), then deleted.
node-overview-before.jpg
137 KiB
node-overview-after.jpg
105 KiB
claude-bot added 1 commit 2026-07-16 23:10:56 +00:00
fix(node-overview): repair 4 dead panels, exclude rbd/lxc devices, nodename legends ac623c1e9c 
- Nodes Online/Down used job="node" but the scrape job is "node-exporter":
  both stats showed No data. Nodes Down additionally gets OR on() vector(0)
  so it renders 0 instead of No data when all nodes are up, and green-at-0/
  red-at-1 thresholds (was permanently red).
- Avg Disk Usage + Disk Usage by Node filtered mountpoint="/" which does not
  exist on Talos (only /var is a real xfs filesystem): both panels were empty.
- Disk I/O summed all block devices including the ~12 Rook-Ceph rbd* mappings
  per node (~438 kB/s vs 8-129 kB/s physical): filter to nvme.*|vd.*.
- Network I/O filter was written for a Docker host (veth/docker/br-) and let
  ~40 Cilium lxc* per-pod interfaces plus kernel pseudo-devices through,
  double-counting pod traffic: filter to eth.* (bond0 is down, 0 slaves).
- Legends showed instance IP:port; join node_uname_info via
  on(instance) group_left(nodename) so panels label k8s-1/2/3. Both join
  sides are wrapped in max by (...) so historical duplicate series from
  pod restarts / the pre-2026-07-16 duplicate-scrape residue cannot break
  the join or rainbow-duplicate long-window views.
- Stat panels reduce lastNotNull instead of mean: show current value, not
  a time-average over the selected window.

All queries verified against live Prometheus before applying.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
k.renner merged commit 21353fd931 into master 2026-07-17 13:17:02 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#289