fix(infrastructure-overview): repair all 5 panels (wrong job, missing labels, Talos mountpoint) #284

2026-07-16T22:50:45Z

claude-bot commented

2026-07-16 22:50:45 +00:00

Review of the Infrastructure Overview dashboard (infrastructure-overview). All 5 panels were broken or degraded — the dashboard appears to have been copied from a Grafana-Cloud-style setup (integrations/unix job, host label) that doesn't match this kube-prometheus-stack + Talos cluster.

Before

After

Review report

#	Panel	Severity	Finding	Status
1	Healthy Hosts	critical	Query used `up{job="integrations/unix"}` — that job does not exist in this Prometheus (node metrics come from job `node-exporter`). Panel showed red "No data".	FIXED
2	Unhealthy Services	major	`count(up == 0)` returns an empty vector when nothing is down, so the healthy state rendered as "No data" instead of 0.	FIXED
3	CPU Usage by Host	critical	`avg by(host)` — the `host` label does not exist on these metrics, so all 3 nodes (and all CPUs) collapsed into one anonymous series labeled "Value".	FIXED
4	Memory Usage by Host	minor	Query worked, but `legendFormat: {{host}}` doesn't exist → legend showed raw label dumps (`{container="node-exporter", endpoint=..., instance="10.2.40.136:9100", ...}`).	FIXED
5	Root Disk Usage by Host	critical	Filtered on `mountpoint="/"` — Talos nodes expose no `/` mountpoint in node_filesystem (root is immutable squashfs). The persistent data disk is `/var` (xfs on /dev/vda5). Panel showed "No data".	FIXED

Changes in detail

Healthy Hosts: count(up{job="node-exporter"} == 1), instant: true. Thresholds changed from red→green(≥1) to red / yellow(1–2) / green(3) so a single down node shows yellow instead of green.
Unhealthy Services: count(up == 0) or vector(0), instant: true — renders an honest 0 when everything is up.
CPU: (100 - avg by (instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) * on(instance) group_left(nodename) node_uname_info, legend {{nodename}} → clean k8s-1/2/3 series.
Memory: same node_uname_info join for readable legends.
Disk: mountpoint="/var" with the nodename join; panel retitled "Data Disk Usage by Host (/var)" to be honest about what is measured.

Not changed (deliberate)

The dashboard has no Ceph/Cilium/Flux panels. I did not add any — this PR is a repair, not a redesign. If you want subsystem coverage here, say so and I'll extend it.
up == 0 counts every scrape target (kubelet, blackbox, authentik, …), not just hosts. Title "Unhealthy Services" matches that behavior, so left as-is.

Verification (live Prometheus, port-forward)

count by (job)(up) — confirmed no integrations/unix job; node-exporter has 3 targets.
Confirmed node_cpu_seconds_total/node_memory_* carry no host label (identity is instance); node_uname_info maps instance → nodename (k8s-1/2/3).
Confirmed node_filesystem_size_bytes{mountpoint="/"} returns nothing; /var (xfs) exists on all 3 nodes.
All 5 replacement queries executed live: Healthy=3, Unhealthy=0, CPU 20–24 %, Mem 27–32 %, /var 49.2 / 62.3 / 65.2 % — matches the after-render.

Review of the **Infrastructure Overview** dashboard (`infrastructure-overview`). All 5 panels were broken or degraded — the dashboard appears to have been copied from a Grafana-Cloud-style setup (`integrations/unix` job, `host` label) that doesn't match this kube-prometheus-stack + Talos cluster. ## Before ![before](https://gitea.renner.uno/attachments/7d35bdc8-6e9c-436a-b378-b6ef01822838) ## After ![after](https://gitea.renner.uno/attachments/acd0c8bb-7d4e-48c6-879d-c70dc4b6af04) ## Review report | # | Panel | Severity | Finding | Status | |---|-------|----------|---------|--------| | 1 | Healthy Hosts | critical | Query used `up{job="integrations/unix"}` — that job does not exist in this Prometheus (node metrics come from job `node-exporter`). Panel showed red "No data". | FIXED | | 2 | Unhealthy Services | major | `count(up == 0)` returns an **empty vector** when nothing is down, so the healthy state rendered as "No data" instead of 0. | FIXED | | 3 | CPU Usage by Host | critical | `avg by(host)` — the `host` label does not exist on these metrics, so all 3 nodes (and all CPUs) collapsed into one anonymous series labeled "Value". | FIXED | | 4 | Memory Usage by Host | minor | Query worked, but `legendFormat: {{host}}` doesn't exist → legend showed raw label dumps (`{container="node-exporter", endpoint=..., instance="10.2.40.136:9100", ...}`). | FIXED | | 5 | Root Disk Usage by Host | critical | Filtered on `mountpoint="/"` — Talos nodes expose **no `/` mountpoint** in node_filesystem (root is immutable squashfs). The persistent data disk is `/var` (xfs on /dev/vda5). Panel showed "No data". | FIXED | ### Changes in detail - **Healthy Hosts**: `count(up{job="node-exporter"} == 1)`, `instant: true`. Thresholds changed from red→green(≥1) to **red / yellow(1–2) / green(3)** so a single down node shows yellow instead of green. - **Unhealthy Services**: `count(up == 0) or vector(0)`, `instant: true` — renders an honest `0` when everything is up. - **CPU**: `(100 - avg by (instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) * on(instance) group_left(nodename) node_uname_info`, legend `{{nodename}}` → clean `k8s-1/2/3` series. - **Memory**: same `node_uname_info` join for readable legends. - **Disk**: `mountpoint="/var"` with the nodename join; panel retitled **"Data Disk Usage by Host (/var)"** to be honest about what is measured. ### Not changed (deliberate) - The dashboard has no Ceph/Cilium/Flux panels. I did not add any — this PR is a repair, not a redesign. If you want subsystem coverage here, say so and I'll extend it. - `up == 0` counts every scrape target (kubelet, blackbox, authentik, …), not just hosts. Title "Unhealthy Services" matches that behavior, so left as-is. ## Verification (live Prometheus, port-forward) - `count by (job)(up)` — confirmed **no** `integrations/unix` job; `node-exporter` has 3 targets. - Confirmed `node_cpu_seconds_total`/`node_memory_*` carry no `host` label (identity is `instance`); `node_uname_info` maps instance → nodename (k8s-1/2/3). - Confirmed `node_filesystem_size_bytes{mountpoint="/"}` returns **nothing**; `/var` (xfs) exists on all 3 nodes. - All 5 replacement queries executed live: Healthy=3, Unhealthy=0, CPU 20–24 %, Mem 27–32 %, /var 49.2 / 62.3 / 65.2 % — matches the after-render.

infrastructure-overview-before.jpg

74 KiB

infrastructure-overview-after.jpg

76 KiB

claude-bot added 1 commit 2026-07-16 22:50:46 +00:00

fix(infrastructure-overview): repair all 5 panels (wrong job, missing labels, Talos mountpoint) ccde43b367

- Healthy Hosts: job 'integrations/unix' does not exist here -> node-exporter;
  thresholds now red/yellow/green(3) so a single down node shows yellow
- Unhealthy Services: 'count(up == 0)' returns empty vector when healthy ->
  'or vector(0)' renders an honest 0; both stats now instant queries
- CPU: 'avg by(host)' collapsed all 3 nodes into one series (no host label) ->
  avg by(instance) joined with node_uname_info for k8s-1/2/3 legend names
- Memory: legend showed raw label dumps ({{host}} does not exist) -> nodename join
- Disk: Talos has no '/' mountpoint in node_filesystem -> use /var (the
  persistent data disk); retitle panel accordingly

k.renner merged commit e8eee7800a into master

2026-07-17 13:15:37 +00:00

k.renner referenced this issue from a commit

2026-07-17 13:15:39 +00:00

Merge pull request 'fix(infrastructure-overview): repair all 5 panels (wrong job, missing labels, Talos mountpoint)' (#284) from claude-bot/dash-infrastructure-overview into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#284