fix(alloy-overview): repoint retired prometheus-legacy datasource, repair host-label legends #293

2026-07-16T23:36:36Z

claude-bot commented

2026-07-16 23:36:36 +00:00

Alloy Agent Overview — review & repair

Before

(screenshot attached below)

After

(screenshot attached below)

Review report

CRITICAL — dashboard 100% dead: retired prometheus-legacy datasource.
All 11 datasource references (9 query panels) pointed at prometheus-legacy, retired 2026-05-09 (documented in dashboards/kustomization.yaml). Every panel rendered blank regardless of query correctness — same root cause as blackbox-overview (#288).
✅ FIXED — repointed all 11 refs to the active prometheus datasource uid.
MAJOR — {{host}} legends reference a label that does not exist.
The job="alloy" self-metrics carry pod/instance/namespace labels but no host label (verified by direct instant queries). All 6 timeseries legends and the Agent Versions table host column would have rendered empty even on the right datasource.
✅ FIXED — Alloy runs as a DaemonSet (one pod per node), so panels now join kube_pod_info (* on(pod) group_left(node) max by (pod, node) (kube_pod_info{namespace="monitoring"})) and legend on {{node}} → k8s-1 / k8s-2 / k8s-3. The max by on the info side guards against duplicate kube_pod_info series on long windows (playbook §19 pattern).
MINOR — Component Evaluation Time (p95) quantile computed over raw per-bucket series.
histogram_quantile(0.95, rate(...bucket[5m])) without aggregation groups implicitly per label set; harmless today but fragile.
✅ FIXED — now histogram_quantile(0.95, sum by (pod, le) (rate(alloy_component_evaluation_seconds_bucket[5m]))) + node join. Verified: ~23 ms p95 per node.
MINOR — Agent Versions table showed scrape-plumbing columns.
After removing the nonexistent host column the table would show container, endpoint, namespace, service, tags noise.
✅ FIXED — table now shows node | pod | version, ordered via indexByName; width override moved from the nonexistent host column to pod.
FINDING (no dashboard change) — coverage: only the 3 in-cluster agents are visible.
alloy_build_info has exactly 3 series, all namespace="monitoring" DaemonSet pods (verified by direct query, not a truncated listing: count by (job) ({__name__=~"alloy_.+"}) returns only job="alloy", 243 series). Not represented:
- the alloy-syslog deployment pods (alloy-syslog-*, present in kube_pod_info) — their self-metrics are not scraped under job="alloy";
- the DNS-VM Alloy instances (dns-servers) — they ship data but do not expose/remote-write their own alloy_* self-metrics into this Prometheus (no matching job in up either).
  "Active Agents = 3" is therefore correct for what is scraped but undercounts the fleet. If you want those agents on this dashboard, they need a ServiceMonitor / prometheus.exporter.self + remote_write on their side — a scrape-config change, not a query change, so I did not fake it here.
  ❓ QUESTION — do you want the alloy-syslog pods and DNS-VM agents scraped so this dashboard covers the whole Alloy fleet?
Note — "Config Load Failures" counts since process start.
sum(alloy_config_load_failures_total) resets on pod restart; currently 0. Left as-is (reasonable for a health tile).

Verification

count(alloy_build_info) → 3 (agents on k8s-1/2/3, all v1.17.1).
Label sets of all 9 metrics inspected via instant queries: no host label anywhere; pod/instance/namespace present.
Every rewritten query run verbatim against live Prometheus before committing — each returns exactly one series per node with plausible values: RSS 250–340 MB, CPU 2.5–3.1 %, network ~90 KB/s, 4 running components/node, p95 eval ~23 ms.
min(alloy_config_last_load_successful) → 1 (OK), sum(alloy_config_load_failures_total) → 0.
Grafana-side check: POST /api/ds/query with count(alloy_build_info) against uid prometheus → HTTP 200 with data (rules out a datasource-uid typo).
JSON validated; diff is surgical (20 lines changed, no reformatting).

# Alloy Agent Overview — review & repair ## Before (screenshot attached below) ## After (screenshot attached below) ## Review report 1. **CRITICAL — dashboard 100% dead: retired `prometheus-legacy` datasource.** All 11 datasource references (9 query panels) pointed at `prometheus-legacy`, retired 2026-05-09 (documented in `dashboards/kustomization.yaml`). Every panel rendered blank regardless of query correctness — same root cause as blackbox-overview (#288). ✅ FIXED — repointed all 11 refs to the active `prometheus` datasource uid. 2. **MAJOR — `{{host}}` legends reference a label that does not exist.** The `job="alloy"` self-metrics carry `pod`/`instance`/`namespace` labels but **no `host` label** (verified by direct instant queries). All 6 timeseries legends and the Agent Versions table `host` column would have rendered empty even on the right datasource. ✅ FIXED — Alloy runs as a DaemonSet (one pod per node), so panels now join `kube_pod_info` (`* on(pod) group_left(node) max by (pod, node) (kube_pod_info{namespace="monitoring"})`) and legend on `{{node}}` → `k8s-1` / `k8s-2` / `k8s-3`. The `max by` on the info side guards against duplicate `kube_pod_info` series on long windows (playbook §19 pattern). 3. **MINOR — Component Evaluation Time (p95) quantile computed over raw per-bucket series.** `histogram_quantile(0.95, rate(...bucket[5m]))` without aggregation groups implicitly per label set; harmless today but fragile. ✅ FIXED — now `histogram_quantile(0.95, sum by (pod, le) (rate(alloy_component_evaluation_seconds_bucket[5m])))` + node join. Verified: ~23 ms p95 per node. 4. **MINOR — Agent Versions table showed scrape-plumbing columns.** After removing the nonexistent `host` column the table would show `container`, `endpoint`, `namespace`, `service`, `tags` noise. ✅ FIXED — table now shows `node | pod | version`, ordered via `indexByName`; width override moved from the nonexistent `host` column to `pod`. 5. **FINDING (no dashboard change) — coverage: only the 3 in-cluster agents are visible.** `alloy_build_info` has exactly 3 series, all `namespace="monitoring"` DaemonSet pods (verified by direct query, not a truncated listing: `count by (job) ({__name__=~"alloy_.+"})` returns only `job="alloy"`, 243 series). Not represented: - the **alloy-syslog** deployment pods (`alloy-syslog-*`, present in `kube_pod_info`) — their self-metrics are not scraped under `job="alloy"`; - the **DNS-VM Alloy instances** (`dns-servers`) — they ship data but do not expose/remote-write their own `alloy_*` self-metrics into this Prometheus (no matching job in `up` either). "Active Agents = 3" is therefore *correct for what is scraped* but undercounts the fleet. If you want those agents on this dashboard, they need a ServiceMonitor / `prometheus.exporter.self` + remote_write on their side — a scrape-config change, not a query change, so I did not fake it here. ❓ QUESTION — do you want the alloy-syslog pods and DNS-VM agents scraped so this dashboard covers the whole Alloy fleet? 6. **Note — "Config Load Failures" counts since process start.** `sum(alloy_config_load_failures_total)` resets on pod restart; currently 0. Left as-is (reasonable for a health tile). ## Verification - `count(alloy_build_info)` → 3 (agents on k8s-1/2/3, all v1.17.1). - Label sets of all 9 metrics inspected via instant queries: no `host` label anywhere; `pod`/`instance`/`namespace` present. - Every rewritten query run verbatim against live Prometheus before committing — each returns exactly one series per node with plausible values: RSS 250–340 MB, CPU 2.5–3.1 %, network ~90 KB/s, 4 running components/node, p95 eval ~23 ms. - `min(alloy_config_last_load_successful)` → 1 (OK), `sum(alloy_config_load_failures_total)` → 0. - Grafana-side check: `POST /api/ds/query` with `count(alloy_build_info)` against uid `prometheus` → HTTP 200 with data (rules out a datasource-uid typo). - JSON validated; diff is surgical (20 lines changed, no reformatting).

alloy-overview-before.jpg

30 KiB

claude-bot added 1 commit 2026-07-16 23:36:36 +00:00

fix(alloy-overview): repoint retired prometheus-legacy datasource, repair host label legends 8519ad875d

Every panel was dead: all 11 datasource refs pointed at the
prometheus-legacy datasource (retired 2026-05-09). Repoint to the
active prometheus datasource.

The alloy job also carries no 'host' label, so all legends and the
Agent Versions table rendered empty labels. Join via kube_pod_info
(pod -> node) so DaemonSet panels are labelled k8s-1/2/3, collapse
the eval-time histogram per pod before quantile, and tidy the
versions table to node/pod/version columns.

All queries verified against live Prometheus (3 agents, one clean
series per node).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

k.renner merged commit 874c3c8e55 into master

2026-07-17 10:59:38 +00:00

k.renner referenced this issue from a commit

2026-07-17 10:59:39 +00:00

Merge pull request 'fix(alloy-overview): repoint retired prometheus-legacy datasource, repair host-label legends' (#293) from claude-bot/dash-alloy-overview into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#293