chore(ansible): retire the monitoring plays for decommissioned hosts #479

2026-08-10T20:11:19Z

claude-bot commented

2026-08-10 20:11:19 +00:00

Prometheus and Loki run in Kubernetes — prometheus-kube-prometheus-stack-prometheus and the Loki release, both up 93 days. The standalone VMs these plays targeted are gone:

prometheus.srv.renner.uno — does not resolve
loki.srv.renner.uno — does not resolve
proxmox_logging — not in the inventory

So these rendered scrape configs, exporter installs and log shippers for hosts that don't exist.

Removed

play	target
`install-prometheus.yml`	`prometheus.srv.renner.uno` (gone)
`install-blackbox-exporter.yml`	`prometheus.srv.renner.uno` (gone)
`install-loki.yml`	`loki.srv.renner.uno` (gone)
`install-log-aggregation.yml`	`proxmox_logging` (no such group)

Roles: prometheus-server, snmp-exporter, pve-exporter, unifi-poller, blackbox-exporter, loki-server, promtail.

Each was consumed only by the plays above; promtail was referenced by nothing at all. Verified no dangling references remain.

plays/update.yml imported install-log-aggregation.yml, so deleting that alone would have broken the fleet update play. The import is removed and update.yml re-syntax-checked.

This also retires #473

That PR changed IPv4 literals to FQDNs in prometheus.yml.j2 and nodes.yml.j2 — a template nothing renders.

Its description explained the instance-label consequences for dashboards and alert rules. That was fiction: no dashboard reads those labels, because that Prometheus doesn't run. When it wouldn't apply I reported it as blocked on Proxmox API credentials; the real reason was that the target host hadn't existed for months, and --list-hosts returning 0 hosts was the evidence I misread.

Worth checking separately — not assumed here

Does anything scrape the PVE node_exporter and chrony_exporter today?

install-node-exporter-pve.yml still installs them on pve_cluster, but the config that pointed a Prometheus at them was in the templates this PR deletes. If that was the only reference, those metrics have been dark since the VM went away — and #473's FQDN change would have masked the question rather than raising it.

I've deliberately left install-node-exporter-pve.yml and the node-exporter/chrony-exporter roles in place, since the answer decides whether they're dead too.

🤖 Generated with Claude Code

Prometheus and Loki run in Kubernetes — `prometheus-kube-prometheus-stack-prometheus` and the Loki release, both up **93 days**. The standalone VMs these plays targeted are gone: - `prometheus.srv.renner.uno` — **does not resolve** - `loki.srv.renner.uno` — **does not resolve** - `proxmox_logging` — **not in the inventory** So these rendered scrape configs, exporter installs and log shippers for hosts that don't exist. ## Removed | play | target | |---|---| | `install-prometheus.yml` | `prometheus.srv.renner.uno` (gone) | | `install-blackbox-exporter.yml` | `prometheus.srv.renner.uno` (gone) | | `install-loki.yml` | `loki.srv.renner.uno` (gone) | | `install-log-aggregation.yml` | `proxmox_logging` (no such group) | Roles: `prometheus-server`, `snmp-exporter`, `pve-exporter`, `unifi-poller`, `blackbox-exporter`, `loki-server`, `promtail`. Each was consumed only by the plays above; `promtail` was referenced by nothing at all. Verified no dangling references remain. `plays/update.yml` imported `install-log-aggregation.yml`, so deleting that alone would have broken the fleet update play. The import is removed and `update.yml` re-syntax-checked. ## This also retires #473 That PR changed IPv4 literals to FQDNs in `prometheus.yml.j2` and `nodes.yml.j2` — a template nothing renders. Its description explained the `instance`-label consequences for dashboards and alert rules. That was fiction: no dashboard reads those labels, because that Prometheus doesn't run. When it wouldn't apply I reported it as blocked on Proxmox API credentials; the real reason was that the target host hadn't existed for months, and `--list-hosts` returning `0 hosts` was the evidence I misread. ## Worth checking separately — not assumed here **Does anything scrape the PVE `node_exporter` and `chrony_exporter` today?** `install-node-exporter-pve.yml` still installs them on `pve_cluster`, but the config that pointed a Prometheus at them was in the templates this PR deletes. If that was the only reference, those metrics have been dark since the VM went away — and #473's FQDN change would have masked the question rather than raising it. I've deliberately left `install-node-exporter-pve.yml` and the `node-exporter`/`chrony-exporter` roles in place, since the answer decides whether they're dead too. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-10 20:11:19 +00:00

chore(ansible): retire the monitoring plays for decommissioned hosts

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

8e5b91e6d6

Prometheus and Loki run in Kubernetes -- statefulsets
prometheus-kube-prometheus-stack-prometheus and the Loki release, both up 93
days. The standalone VMs these plays targeted are gone: neither
prometheus.srv.renner.uno nor loki.srv.renner.uno resolves at all, and the
proxmox_logging group install-log-aggregation.yml targets is not in the
inventory.

So these rendered scrape configs, exporter installs and log shippers for
hosts that do not exist. Removed:

  plays/install-prometheus.yml          -> prometheus.srv.renner.uno (gone)
  plays/install-blackbox-exporter.yml   -> prometheus.srv.renner.uno (gone)
  plays/install-loki.yml                -> loki.srv.renner.uno (gone)
  plays/install-log-aggregation.yml     -> proxmox_logging (no such group)
  roles/prometheus-server, snmp-exporter, pve-exporter, unifi-poller,
  roles/blackbox-exporter, loki-server, promtail

Each role was consumed only by the plays above; promtail was referenced by
nothing at all. Verified no dangling references remain.

plays/update.yml imported install-log-aggregation.yml, so deleting it alone
would have broken the fleet update play. That import is removed and
update.yml re-syntax-checked.

This also retires #473, which changed IPv4 literals to FQDNs in
prometheus.yml.j2 and nodes.yml.j2. That was a change to a template nothing
renders. Its PR body described the instance-label consequences for
dashboards and alert rules, which was fiction -- no dashboard reads those
labels. When it would not apply I reported it as blocked on Proxmox API
credentials; the actual reason was that the target host had not existed for
months, and `--list-hosts` returning 0 was the evidence I misread.

Worth checking separately, not assumed here: whether anything scrapes the
PVE node_exporter and chrony_exporter today. install-node-exporter-pve.yml
still installs them on pve_cluster, but the config that pointed a Prometheus
at them was in the templates just deleted. If that was the only reference,
those metrics have been dark since the VM went away -- and the FQDN change
in #473 would have masked the question rather than raising it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-10 20:15:06 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 8e5b91e6d677d43b0f8f6e970b86f57741e198af.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Pure-deletion chore PR (0 additions, 1691 deletions across 42 files): removes four Ansible plays (install-prometheus.yml, install-blackbox-exporter.yml, install-loki.yml, install-log-aggregation.yml) and the seven roles they alone consumed (prometheus-server, snmp-exporter, pve-exporter, unifi-poller, blackbox-exporter, loki-server, promtail), and drops the now-dangling import_playbook: install-log-aggregation.yml line from plays/update.yml. I verified the mechanics of the deletion are clean; my one substantive finding is about the PR's stated rationale, not the diff itself.

No blocking issues.

🟡 Suggestions

The PR's core premise checks out, but other live code still targets the same "dead" host — worth a companion cleanup. The PR asserts prometheus.srv.renner.uno / loki.srv.renner.uno are gone and DNS doesn't resolve. kubernetes/apps/monitoring/dashboards/kustomization.yaml:19-21 corroborates this precisely: "The prometheus-legacy datasource (LXC 227) was retired with the legacy stack on 2026-05-09." That matches the PR's "in-cluster Prometheus up 93 days" framing almost to the day. However, several files this PR does not touch still hard-code these hostnames as live endpoints:
- ansible/roles/monitoring-agent/defaults/main.yml:19-20 — loki_endpoint/prometheus_remote_write_endpoint default to the dead hosts.
- ansible/plays/common.yml:16-17 — same, and this role (monitoring-agent) is applied to hosts: all:!pve:!pbs:!proxmox_noansible:!proxmox_nodes:!pve_cluster, i.e. nearly the whole fleet.
- ansible/plays/install-pve-monitoring.yml:12-13 — same, targeting pve_cluster.
- ansible/plays/install-postgres-monitoring.yml:42 — hosts: prometheus.srv.renner.uno is the literal target of a play; if the hostname truly doesn't resolve/isn't inventoried, this play currently matches 0 hosts and is dead.
- kubernetes/apps/monitoring/helmrelease-prometheus.yaml:218-228 — a live Grafana datasource (prometheus-legacy) still points at http://prometheus.srv.renner.uno:9090, with a comment saying it "Retires at global cut-over" — which reads as not-yet-happened, in apparent tension with the kustomization.yaml comment above saying it already retired 2026-05-09.
None of this is caused by this PR — these files were already stale before this change, and "no dangling references" in the PR body is accurate for the specific roles/plays being deleted. But the PR's narrative ("standalone VMs... are gone") is incomplete: other automation still assumes the same host is alive, and one play (install-postgres-monitoring.yml) is silently broken by the same fact this PR is otherwise cleaning up. This fits the same "worth checking separately" spirit the PR body already applies to install-node-exporter-pve.yml — worth a follow-up chore to either repoint or delete monitoring-agent's defaults, common.yml/install-pve-monitoring.yml's endpoint vars, install-postgres-monitoring.yml, and the leftover Grafana prometheus-legacy datasource.

Tests

No test suite applies here (Ansible YAML deletions); the PR body says update.yml was re-syntax-checked after removing the dangling import, which is the right verification for this kind of change. I independently confirmed: the deleted play/role names have zero remaining references anywhere under ansible/, and plays/update.yml no longer imports the removed play.

Recommendation

Looks reasonable — no blocking issues; the suggestion above is optional follow-up, not a reason to hold this PR.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `8e5b91e6d677d43b0f8f6e970b86f57741e198af`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Pure-deletion chore PR (0 additions, 1691 deletions across 42 files): removes four Ansible plays (`install-prometheus.yml`, `install-blackbox-exporter.yml`, `install-loki.yml`, `install-log-aggregation.yml`) and the seven roles they alone consumed (`prometheus-server`, `snmp-exporter`, `pve-exporter`, `unifi-poller`, `blackbox-exporter`, `loki-server`, `promtail`), and drops the now-dangling `import_playbook: install-log-aggregation.yml` line from `plays/update.yml`. I verified the mechanics of the deletion are clean; my one substantive finding is about the PR's stated rationale, not the diff itself. No blocking issues. ## 🟡 Suggestions - **The PR's core premise checks out, but other live code still targets the same "dead" host — worth a companion cleanup.** The PR asserts `prometheus.srv.renner.uno` / `loki.srv.renner.uno` are gone and DNS doesn't resolve. `kubernetes/apps/monitoring/dashboards/kustomization.yaml:19-21` corroborates this precisely: *"The `prometheus-legacy` datasource (LXC 227) was retired with the legacy stack on 2026-05-09."* That matches the PR's "in-cluster Prometheus up 93 days" framing almost to the day. However, several files this PR does **not** touch still hard-code these hostnames as live endpoints: - `ansible/roles/monitoring-agent/defaults/main.yml:19-20` — `loki_endpoint`/`prometheus_remote_write_endpoint` default to the dead hosts. - `ansible/plays/common.yml:16-17` — same, and this role (`monitoring-agent`) is applied to `hosts: all:!pve:!pbs:!proxmox_noansible:!proxmox_nodes:!pve_cluster`, i.e. nearly the whole fleet. - `ansible/plays/install-pve-monitoring.yml:12-13` — same, targeting `pve_cluster`. - `ansible/plays/install-postgres-monitoring.yml:42` — `hosts: prometheus.srv.renner.uno` is the literal target of a play; if the hostname truly doesn't resolve/isn't inventoried, this play currently matches 0 hosts and is dead. - `kubernetes/apps/monitoring/helmrelease-prometheus.yaml:218-228` — a live Grafana datasource (`prometheus-legacy`) still points at `http://prometheus.srv.renner.uno:9090`, with a comment saying it "Retires at global cut-over" — which reads as not-yet-happened, in apparent tension with the kustomization.yaml comment above saying it already retired 2026-05-09. None of this is caused by this PR — these files were already stale before this change, and "no dangling references" in the PR body is accurate for the specific roles/plays being deleted. But the PR's narrative ("standalone VMs... are gone") is incomplete: other automation still assumes the same host is alive, and one play (`install-postgres-monitoring.yml`) is silently broken by the same fact this PR is otherwise cleaning up. This fits the same "worth checking separately" spirit the PR body already applies to `install-node-exporter-pve.yml` — worth a follow-up chore to either repoint or delete `monitoring-agent`'s defaults, `common.yml`/`install-pve-monitoring.yml`'s endpoint vars, `install-postgres-monitoring.yml`, and the leftover Grafana `prometheus-legacy` datasource. ## Tests No test suite applies here (Ansible YAML deletions); the PR body says `update.yml` was re-syntax-checked after removing the dangling import, which is the right verification for this kind of change. I independently confirmed: the deleted play/role names have zero remaining references anywhere under `ansible/`, and `plays/update.yml` no longer imports the removed play. ## Recommendation Looks reasonable — no blocking issues; the suggestion above is optional follow-up, not a reason to hold this PR.

k.renner merged commit 0581bc334a into master

2026-08-12 16:30:31 +00:00

k.renner referenced this issue from a commit

2026-08-12 16:30:33 +00:00

Merge pull request 'chore(ansible): retire the monitoring plays for decommissioned hosts' (#479) from retire-dead-monitoring into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#479