refactor(prometheus): address the PVE nodes by name in scrape targets #473

2026-08-07T17:53:18Z

claude-bot commented

2026-08-07 17:53:18 +00:00

Follow-up to #472, which changed the inventory but left the same three IPv4 literals in the monitoring config and a runbook. Raised in review on that PR.

`prometheus.yml.j2`

pve job targets → FQDNs
chrony job targets → FQDNs, restructured to one static_config per host so each carries its own hostname label

The chrony restructure is the substantive part, not cosmetics. The previous form listed all three targets together and recovered the hostname in relabel_configs by regex-matching __address__ against each node's IP:

- source_labels: [__address__]
  regex: '10\.2\.40\.201:.*'
  target_label: hostname
  replacement: 'jupiter'

That silently coupled the labels to the addressing. A find-and-replace on the target addresses alone would have left every regex unmatched and the hostname label simply absent — with the scrape still succeeding and nothing failing. Per-host static_configs remove the coupling entirely.

`nodes.yml.j2`

Targets → FQDNs. Already one entry per host with an explicit hostname label, so nothing else needed.

`claude-workstation/RUNBOOK.md`

The kubeconfig command pointed --server at https://10.2.40.201:6443. That is jupiter, a Proxmox host — not a control-plane node — stale from when k8s ran as VMs on it.

The cluster endpoint is https://k8s.renner.uno:6443 (kubernetes/talos/patches/cluster.yaml). The name is also what the API server certificate carries as a SAN, so an address there fails TLS verification even when reachable. The documented command was very likely broken as written, not merely inconsistent.

Deliberately not changed

kubernetes/apps/pve/endpointslice.yaml holds the same three addresses. An EndpointSlice takes literal addresses — hostnames are not valid in that field — so those are correct as they are. Flagging it because a sweep for the literals will surface it.

Behaviour change worth noting before merge

This changes the instance label on the pve job from an address to a name: the relabel chain feeds __address__ → __param_target → instance. Anything selecting on the old value (dashboards, alert rules, recording rules) needs updating, and existing series will not stitch to the new ones.

The chrony and node jobs already label by hostname, so those are unaffected.

🤖 Generated with Claude Code

Follow-up to #472, which changed the inventory but left the same three IPv4 literals in the monitoring config and a runbook. Raised in review on that PR. ## `prometheus.yml.j2` - **pve job** targets → FQDNs - **chrony job** targets → FQDNs, restructured to one `static_config` per host so each carries its own `hostname` label **The chrony restructure is the substantive part, not cosmetics.** The previous form listed all three targets together and recovered the hostname in `relabel_configs` by regex-matching `__address__` against each node's IP: ```yaml - source_labels: [__address__] regex: '10\.2\.40\.201:.*' target_label: hostname replacement: 'jupiter' ``` That silently coupled the labels to the addressing. A find-and-replace on the target addresses alone would have left every regex unmatched and the `hostname` label simply **absent** — with the scrape still succeeding and nothing failing. Per-host `static_configs` remove the coupling entirely. ## `nodes.yml.j2` Targets → FQDNs. Already one entry per host with an explicit `hostname` label, so nothing else needed. ## `claude-workstation/RUNBOOK.md` The kubeconfig command pointed `--server` at `https://10.2.40.201:6443`. That is **jupiter, a Proxmox host** — not a control-plane node — stale from when k8s ran as VMs on it. The cluster endpoint is `https://k8s.renner.uno:6443` (`kubernetes/talos/patches/cluster.yaml`). The name is also what the API server certificate carries as a SAN, so an address there fails TLS verification even when reachable. **The documented command was very likely broken as written**, not merely inconsistent. ## Deliberately not changed `kubernetes/apps/pve/endpointslice.yaml` holds the same three addresses. An EndpointSlice takes literal addresses — hostnames are not valid in that field — so those are correct as they are. Flagging it because a sweep for the literals will surface it. ## Behaviour change worth noting before merge This changes the `instance` label on the **pve** job from an address to a name: the relabel chain feeds `__address__` → `__param_target` → `instance`. Anything selecting on the old value (dashboards, alert rules, recording rules) needs updating, and existing series will not stitch to the new ones. The chrony and node jobs already label by `hostname`, so those are unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-07 17:53:18 +00:00

refactor(prometheus): address the PVE nodes by name in scrape targets

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

17c12fe13f

Follow-up to #472, which changed the inventory but left the same three IPv4
literals in the monitoring config and a runbook. Raised in review there.

prometheus.yml.j2:
  - pve job targets -> FQDNs
  - chrony job targets -> FQDNs, and restructured to one static_config per
    host so each carries its own hostname label

The chrony restructure is the substantive part. The previous form listed all
three targets together and recovered the hostname in relabel_configs by
regex-matching __address__ against each node's IP:

    - source_labels: [__address__]
      regex: '10\.2\.40\.201:.*'
      target_label: hostname
      replacement: 'jupiter'

That silently coupled the labels to the addressing. Swapping a target for a
name would have left every regex unmatched and the hostname label simply
absent, with the scrape still succeeding and nothing failing -- a
find-and-replace on the addresses alone would have broken the labels
quietly. Per-host static_configs remove the coupling entirely.

nodes.yml.j2: targets -> FQDNs. Already one entry per host with an explicit
hostname label, so nothing else to change.

claude-workstation/RUNBOOK.md: the kubeconfig command pointed --server at
https://10.2.40.201:6443. That is jupiter, a Proxmox host, not a control
plane node -- stale from when k8s ran as VMs on it. The cluster's endpoint
is https://k8s.renner.uno:6443 (kubernetes/talos/patches/cluster.yaml), and
the name is also what the API server certificate carries as a SAN, so an
address there fails TLS verification even when reachable. The documented
command was very likely broken as written.

Deliberately NOT changed: kubernetes/apps/pve/endpointslice.yaml. An
EndpointSlice takes literal addresses; hostnames are not valid in that
field, so those three are correct as they are.

Note this changes the `instance` label on the pve job from an address to a
name, since the relabel chain feeds __address__ into __param_target and on
to instance. Anything selecting on the old value needs updating.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-07 17:55:27 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 17c12fe13f882bde66187def3dab285560aa0857.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Follow-up to #472: replaces the three PVE-node IPv4 literals with FQDNs in prometheus.yml.j2 and nodes.yml.j2, restructures the chrony job from one static_config + IP-matching relabel_configs into one static_config per host, and fixes a stale --server IP in the claude-workstation RUNBOOK that pointed at a Proxmox host instead of the k8s API endpoint. Verified against the checkout: the FQDNs match the ansible_host values already used elsewhere (ansible/inventory/pve_hosts.yml:14-16), k8s.renner.uno is a real certSAN and the cluster's documented endpoint (kubernetes/talos/patches/cluster.yaml:6,44), and 10.2.40.201 is not a SAN — confirming the old RUNBOOK command would indeed fail TLS verification. The pve EndpointSlice was correctly left untouched, since discovery.k8s.io/v1 addresses must be literal IPs. No blocking issues found.

🟡 Suggestions

ansible/roles/prometheus-server/templates/prometheus.yml.j2:47-59 (pve job) — the PR body flags a real, unaddressed consequence: this changes the instance label on the pve job's series from an IP to an FQDN (via __address__ → __param_target → instance). I couldn't find any dashboards/alert/recording rules in this repo referencing the old instance value for job="pve" — those likely live in Grafana rather than in-repo, so I can't confirm the blast radius. Worth double-checking Grafana dashboards/alert rules for job="pve" selectors before or right after merge, since old and new series won't stitch together (a gap in the pve panels around the deploy).

Tests

No test suite applies here — these are Jinja2-templated Prometheus/Ansible config and a markdown runbook, consistent with how the rest of the repo handles this kind of change (no CI-checked test coverage for template content beyond what deploy-time linting/rendering catches). Not a gap specific to this PR.

Recommendation

Looks reasonable — no blocking issues; the instance-label comment above is worth a quick check but isn't something this diff needs to fix.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `17c12fe13f882bde66187def3dab285560aa0857`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Follow-up to #472: replaces the three PVE-node IPv4 literals with FQDNs in `prometheus.yml.j2` and `nodes.yml.j2`, restructures the `chrony` job from one `static_config` + IP-matching `relabel_configs` into one `static_config` per host, and fixes a stale `--server` IP in the claude-workstation RUNBOOK that pointed at a Proxmox host instead of the k8s API endpoint. Verified against the checkout: the FQDNs match the `ansible_host` values already used elsewhere (`ansible/inventory/pve_hosts.yml:14-16`), `k8s.renner.uno` is a real certSAN and the cluster's documented endpoint (`kubernetes/talos/patches/cluster.yaml:6,44`), and `10.2.40.201` is *not* a SAN — confirming the old RUNBOOK command would indeed fail TLS verification. The `pve` EndpointSlice was correctly left untouched, since `discovery.k8s.io/v1` addresses must be literal IPs. No blocking issues found. ## 🟡 Suggestions - `ansible/roles/prometheus-server/templates/prometheus.yml.j2:47-59` (pve job) — the PR body flags a real, unaddressed consequence: this changes the `instance` label on the `pve` job's series from an IP to an FQDN (via `__address__` → `__param_target` → `instance`). I couldn't find any dashboards/alert/recording rules in this repo referencing the old `instance` value for `job="pve"` — those likely live in Grafana rather than in-repo, so I can't confirm the blast radius. Worth double-checking Grafana dashboards/alert rules for `job="pve"` selectors before or right after merge, since old and new series won't stitch together (a gap in the `pve` panels around the deploy). ## Tests No test suite applies here — these are Jinja2-templated Prometheus/Ansible config and a markdown runbook, consistent with how the rest of the repo handles this kind of change (no CI-checked test coverage for template content beyond what deploy-time linting/rendering catches). Not a gap specific to this PR. ## Recommendation Looks reasonable — no blocking issues; the instance-label comment above is worth a quick check but isn't something this diff needs to fix.

k.renner merged commit 5b03d6b7af into master

2026-08-08 05:11:30 +00:00

k.renner referenced this issue from a commit

2026-08-08 05:11:31 +00:00

Merge pull request 'refactor(prometheus): address the PVE nodes by name in scrape targets' (#473) from prometheus-pve-hostnames into master

claude-bot referenced this issue from a commit

2026-08-10 20:10:52 +00:00

chore(ansible): retire the monitoring plays for decommissioned hosts

claude-bot referenced this pull request

2026-08-10 20:11:19 +00:00

chore(ansible): retire the monitoring plays for decommissioned hosts #479

Sign in to join this conversation.