konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 5 Packages Projects Releases 1 Wiki Activity

fix(postgresql-overview): scrape CNPG metrics and rewrite dead pg_* queries #286

Merged
k.renner merged 1 commits from claude-bot/dash-postgresql-overview into master 2026-07-17 13:17:59 +00:00
Conversation 0 Commits 1 Files Changed 3
+106 -30
claude-bot commented 2026-07-16 23:05:59 +00:00
Member
Copy Link
Copy Source

PostgreSQL Overview — dashboard was 100% dead; root cause fixed

Before

before

After (temp-preview render)

after

Note on the After shot: it still shows "No data" — necessarily. The root cause is that no PostgreSQL metrics are scraped at all, and the fix for that (the new PodMonitors) only takes effect when this PR merges and Flux applies it. I did not apply the PodMonitors out-of-band to the live cluster (the action was blocked, and it belongs in Git anyway). What the After shot demonstrates is the corrected layout/panels; every query was verified as described under Verification. ~2 minutes after merge the panels will populate.

Review report

Critical — the whole dashboard queries metrics that have never existed

FINDING (critical): All 11 panels use classic postgres_exporter metrics (pg_up, pg_stat_activity_count, pg_database_size_bytes, pg_locks_count, pg_stat_database_*, pg_replication_lag_seconds). This Prometheus contains zero pg_* series — there is no postgres_exporter deployed and never was. Postgres runs as four CloudNativePG clusters (ak-pg/authentik, im-pg/immich, gf-pg/grafana, pprl-pg/paperless). Their instance pods expose a built-in exporter on port 9187 with cnpg_* metrics — but nothing scrapes it: all four clusters have enablePodMonitor: false and no PodMonitor exists.

✅ FIXED (two parts):

  1. Scraping: added kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml — one PodMonitor per CNPG cluster, labelled release: kube-prometheus-stack. That label matters: this Prometheus has podMonitorSelector: {release: kube-prometheus-stack}, so simply flipping enablePodMonitor: true in the Cluster specs would generate PodMonitors that Prometheus ignores. The monitors also copy the pod role label (primary/replica) onto the series for future use.
  2. Queries: rewrote every panel to the cnpg_* metrics the exporter actually exposes (each one verified against the live pods' /metrics endpoints — see Verification).

Per-panel changes

Panel Was Now
Database Status pg_up (nonexistent) Instances: cnpg_collector_up per pod, Up/Down mapping — all 6 instance pods visible individually
Active Connections sum(pg_stat_activity_count{state="active"}) Connections: sum(cnpg_backends_total); thresholds 200/320 (ak-pg alone allows 400)
Total Database Size sum(pg_database_size_bytes) sum(max by (namespace, datname) (...)) excluding template*/postgres — the max by dedupes the 2-instance clusters (primary + replica both report every DB's size; a plain sum would double-count ak-pg and pprl-pg)
Active Locks pg_locks_count (nonexistent; CNPG's default query set has no lock metric) Max Replication Lag: max(cnpg_pg_replication_lag), thresholds 10 s/60 s — more useful here anyway since two clusters run a sync standby
Transactions/sec raw sum over nonexistent counters rate() over cnpg_pg_stat_database_xact_commit/_rollback, primary-only via and on (namespace, pod) (cnpg_pg_replication_in_recovery == 0) so replica read-only transactions (mostly the exporter itself) don't inflate the number
Connection States 3 targets on nonexistent metric sum by (namespace, state) (cnpg_backends_total) — per cluster and per state, so the four instances are distinguishable
Transaction Rate per-datname rates same, on cnpg_*, primary-only, template*/postgres/empty datname excluded
Cache Hit Ratio lifetime-total ratio with a +0.001 fudge — a lifetime ratio is nearly constant and hides regressions rate-based 5 m ratio per database, primary-only; y-axis clamped 0–100 %. Idle DBs now show a gap (honest) instead of a fake number
Database Sizes per-datname, double-counted on 2-instance clusters max by (namespace, datname), system DBs excluded
Tuple Operations 4×4 per-datname series (16 lines) 4 clean totals (Fetched/Inserted/Updated/Deleted), primary-only
Replication & Transaction Health both metrics nonexistent max by (namespace) (cnpg_pg_replication_lag) + max by (namespace) (cnpg_backends_max_tx_duration_seconds)

Also: all 5 stat panels now use instant queries, so a dead instance shows No data instead of silently freezing its last value (cross-dashboard lesson from the home-overview review). Panels distinguish instances via namespace (each CNPG cluster lives in its own namespace), which is a label Prometheus guarantees on every scraped series — I deliberately did not build queries on relabel-produced labels I could not verify pre-merge.

Not fixed / follow-up ideas

  • ⚠️ NOT FIXED — WAL archiving/backup health not panelled. The exporter exposes cnpg_pg_stat_archiver_* (e.g. failed_count was 29 on ak-pg-1) and barman-cloud backup timestamps. Worth a follow-up panel or, better, alert rules. Left out to keep this PR reviewable.
  • ❓ QUESTION — dedicated per-cluster drill-down? If you want per-cluster deep dives (WAL, checkpoints, bgwriter, per-table stats), the upstream CNPG Grafana dashboard could be added as a second dashboard; this one stays the fleet overview. Say the word and I'll prepare it.

Verification

  • curl http://prometheus:9090/api/v1/label/__name__/values → 0 metrics matching ^pg_|^cnpg_; no postgres-related scrape job → confirmed root cause.
  • kubectl get clusters.postgresql.cnpg.io -A → 4 healthy clusters; all with enablePodMonitor: false; kubectl get podmonitors -A → only velero's.
  • Prometheus spec: podMonitorSelector: {matchLabels: {release: kube-prometheus-stack}}, podMonitorNamespaceSelector: {} → cross-namespace OK, release label required (pattern confirmed against the working immich/velero monitors).
  • Every metric name and its label set verified against the live exporters via kubectl get --raw .../pods/<pod>:9187/proxy/metrics on ak-pg-1, im-pg-1, gf-pg-1, pprl-pg-3 (databases: authentik 1.29 GB, immich 610 MB, grafana 21 MB, paperless 174 MB — the Total Database Size stat should show ≈2.1 GiB after merge).
  • Pod port name metrics (9187) and selector label cnpg.io/cluster verified on the live pods.
  • All 16 new PromQL expressions executed against live Prometheus: all parse and return success (0 series, as expected pre-merge).
  • Dashboard JSON validated (json.load, 11 panels); PodMonitor YAML validated (4 docs).
# PostgreSQL Overview — dashboard was 100% dead; root cause fixed ## Before ![before](https://gitea.renner.uno/attachments/fb8e589e-9b67-4b1d-b6f6-fb93e2523dec) ## After (temp-preview render) ![after](https://gitea.renner.uno/attachments/8bf687e3-5c11-4633-a978-022aef64e9cc) **Note on the After shot:** it still shows "No data" — necessarily. The root cause is that **no PostgreSQL metrics are scraped at all**, and the fix for that (the new PodMonitors) only takes effect when this PR merges and Flux applies it. I did not apply the PodMonitors out-of-band to the live cluster (the action was blocked, and it belongs in Git anyway). What the After shot demonstrates is the corrected layout/panels; every query was verified as described under *Verification*. ~2 minutes after merge the panels will populate. ## Review report ### Critical — the whole dashboard queries metrics that have never existed **FINDING (critical):** All 11 panels use classic **postgres_exporter** metrics (`pg_up`, `pg_stat_activity_count`, `pg_database_size_bytes`, `pg_locks_count`, `pg_stat_database_*`, `pg_replication_lag_seconds`). This Prometheus contains **zero** `pg_*` series — there is no postgres_exporter deployed and never was. Postgres runs as four **CloudNativePG** clusters (`ak-pg`/authentik, `im-pg`/immich, `gf-pg`/grafana, `pprl-pg`/paperless). Their instance pods expose a built-in exporter on port 9187 with `cnpg_*` metrics — but **nothing scrapes it**: all four clusters have `enablePodMonitor: false` and no PodMonitor exists. **✅ FIXED (two parts):** 1. **Scraping:** added `kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml` — one PodMonitor per CNPG cluster, labelled `release: kube-prometheus-stack`. That label matters: this Prometheus has `podMonitorSelector: {release: kube-prometheus-stack}`, so simply flipping `enablePodMonitor: true` in the Cluster specs would generate PodMonitors that Prometheus **ignores**. The monitors also copy the pod `role` label (primary/replica) onto the series for future use. 2. **Queries:** rewrote every panel to the `cnpg_*` metrics the exporter actually exposes (each one verified against the live pods' `/metrics` endpoints — see Verification). ### Per-panel changes | Panel | Was | Now | |---|---|---| | Database Status | `pg_up` (nonexistent) | **Instances**: `cnpg_collector_up` per pod, Up/Down mapping — all 6 instance pods visible individually | | Active Connections | `sum(pg_stat_activity_count{state="active"})` | **Connections**: `sum(cnpg_backends_total)`; thresholds 200/320 (ak-pg alone allows 400) | | Total Database Size | `sum(pg_database_size_bytes)` | `sum(max by (namespace, datname) (...))` excluding `template*`/`postgres` — the `max by` **dedupes the 2-instance clusters** (primary + replica both report every DB's size; a plain `sum` would double-count ak-pg and pprl-pg) | | Active Locks | `pg_locks_count` (nonexistent; CNPG's default query set has **no lock metric**) | **Max Replication Lag**: `max(cnpg_pg_replication_lag)`, thresholds 10 s/60 s — more useful here anyway since two clusters run a sync standby | | Transactions/sec | raw sum over nonexistent counters | `rate()` over `cnpg_pg_stat_database_xact_commit/_rollback`, **primary-only** via `and on (namespace, pod) (cnpg_pg_replication_in_recovery == 0)` so replica read-only transactions (mostly the exporter itself) don't inflate the number | | Connection States | 3 targets on nonexistent metric | `sum by (namespace, state) (cnpg_backends_total)` — per cluster **and** per state, so the four instances are distinguishable | | Transaction Rate | per-datname rates | same, on `cnpg_*`, primary-only, `template*`/`postgres`/empty datname excluded | | Cache Hit Ratio | lifetime-total ratio with a `+0.001` fudge — a lifetime ratio is nearly constant and hides regressions | **rate-based** 5 m ratio per database, primary-only; y-axis clamped 0–100 %. Idle DBs now show a gap (honest) instead of a fake number | | Database Sizes | per-datname, double-counted on 2-instance clusters | `max by (namespace, datname)`, system DBs excluded | | Tuple Operations | 4×4 per-datname series (16 lines) | 4 clean totals (Fetched/Inserted/Updated/Deleted), primary-only | | Replication & Transaction Health | both metrics nonexistent | `max by (namespace) (cnpg_pg_replication_lag)` + `max by (namespace) (cnpg_backends_max_tx_duration_seconds)` | Also: all 5 stat panels now use **instant queries**, so a dead instance shows *No data* instead of silently freezing its last value (cross-dashboard lesson from the home-overview review). Panels distinguish instances via `namespace` (each CNPG cluster lives in its own namespace), which is a label Prometheus guarantees on every scraped series — I deliberately did not build queries on relabel-produced labels I could not verify pre-merge. ### Not fixed / follow-up ideas - **⚠️ NOT FIXED — WAL archiving/backup health not panelled.** The exporter exposes `cnpg_pg_stat_archiver_*` (e.g. `failed_count` was 29 on ak-pg-1) and barman-cloud backup timestamps. Worth a follow-up panel or, better, alert rules. Left out to keep this PR reviewable. - **❓ QUESTION — dedicated per-cluster drill-down?** If you want per-cluster deep dives (WAL, checkpoints, bgwriter, per-table stats), the upstream CNPG Grafana dashboard could be added as a second dashboard; this one stays the fleet overview. Say the word and I'll prepare it. ## Verification - `curl http://prometheus:9090/api/v1/label/__name__/values` → **0** metrics matching `^pg_|^cnpg_`; no postgres-related scrape job → confirmed root cause. - `kubectl get clusters.postgresql.cnpg.io -A` → 4 healthy clusters; all with `enablePodMonitor: false`; `kubectl get podmonitors -A` → only velero's. - Prometheus spec: `podMonitorSelector: {matchLabels: {release: kube-prometheus-stack}}`, `podMonitorNamespaceSelector: {}` → cross-namespace OK, release label required (pattern confirmed against the working immich/velero monitors). - Every metric name and its label set verified against the live exporters via `kubectl get --raw .../pods/<pod>:9187/proxy/metrics` on ak-pg-1, im-pg-1, gf-pg-1, pprl-pg-3 (databases: authentik 1.29 GB, immich 610 MB, grafana 21 MB, paperless 174 MB — the Total Database Size stat should show ≈2.1 GiB after merge). - Pod port name `metrics` (9187) and selector label `cnpg.io/cluster` verified on the live pods. - All 16 new PromQL expressions executed against live Prometheus: all parse and return success (0 series, as expected pre-merge). - Dashboard JSON validated (`json.load`, 11 panels); PodMonitor YAML validated (4 docs).
postgresql-overview-before.jpg
38 KiB
postgresql-overview-after.jpg
38 KiB
claude-bot added 1 commit 2026-07-16 23:05:59 +00:00
fix(postgresql-overview): scrape CNPG metrics and rewrite dead pg_* queries c7e9fc8bb6 
The dashboard was entirely dead: every panel queried postgres_exporter
metrics (pg_up, pg_stat_activity_count, ...) that have never existed in
this Prometheus. Postgres runs as four CloudNativePG clusters (ak-pg,
im-pg, gf-pg, pprl-pg) whose built-in exporter on :9187 was not scraped
at all (enablePodMonitor is false and no PodMonitor existed).

- add PodMonitors for the four CNPG clusters in monitoring-rules,
  labelled release=kube-prometheus-stack so the Prometheus
  podMonitorSelector actually picks them up (CNPG's own
  enablePodMonitor would generate unlabelled PodMonitors that this
  Prometheus ignores)
- rewrite all dashboard queries to the cnpg_* metrics the exporter
  actually exposes (verified against the pods' /metrics endpoints)
- dedupe multi-instance clusters (max by namespace/datname for sizes,
  primary-only filter via cnpg_pg_replication_in_recovery == 0 for
  per-database counters)
- replace the Active Locks stat (no lock metric in the CNPG default
  queries) with Max Replication Lag
- instant queries on all stat panels so dead instances show No data
  instead of a stale last value

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
k.renner merged commit bf534e3f45 into master 2026-07-17 13:17:59 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#286