fix: [alert] TargetDown (pve-node) (#573) #577

2026-08-25T18:48:49Z

issue-fix-bot commented

2026-08-25 18:48:49 +00:00

What this PR does

Read this first: this PR does not stop TargetDown (pve-node) from firing, and it is not meant to. The alert in #573 is genuinely not repo-fixable — jupiter is powered down mid-conversion, and no manifest change makes an offline host answer. What I did find, while verifying the issue's claims, is a real and currently-active defect in one of the two files the issue named: kubernetes/apps/pve/endpointslice.yaml hardcodes ready: true for jupiter's dead .201 address. Because that EndpointSlice is manually managed, Kubernetes health-checks nothing in it, so kube-proxy and Traefik keep load balancing onto a host that is unreachable. That silently broke the pve cluster scrape job (a different job from the one that alerted) and the pve.renner.uno web UI for a large fraction of requests. This PR marks that one endpoint not-ready, which is the supported way to drain a manual endpoint.

The issue explicitly asserted "nothing about the scrape config (helmrelease-prometheus.yaml, endpointslice.yaml) is wrong." That assessment is incorrect, and the evidence is below.

Changes

kubernetes/apps/pve/endpointslice.yaml — set jupiter's endpoint conditions.ready to false (was true) and document why. Removes the dead 10.2.40.201 from the pve Service's load-balancing set. The address, hostname and comment history are left intact so restoring it is a one-word change.

No other file touched. In particular I did not remove jupiter.renner.uno from the pve-node target list — see Notes.

Verification

Everything below was observed in command output during this run.

./.ci-diag/kubectl get nodes → only k8s-2 and k8s-3 Ready; k8s-1 absent. Confirms the issue's claim.
.../query?query=up{job="pve-node"} → jupiter.renner.uno = 0, neptune = 1, saturn = 1. Confirms the issue's claim.
.../targets?state=any&scrapePool=pve → the cluster job is also health: "down", lastError: "server returned HTTP status 500 INTERNAL SERVER ERROR". The issue did not notice this.
./.ci-diag/kubectl logs -n monitoring pve-exporter-b8b97bcd5-rwbtg → two distinct failures, which is the key evidence:
- HTTPSConnectionPool(host='jupiter.renner.uno', port=8006) ... [Errno 113] Host is unreachable — the pve-node job, expected while jupiter is down.
- HTTPSConnectionPool(host='pve.apps.svc.cluster.local', port=8006) ... [Errno 113] Host is unreachable — the pve cluster job, i.e. the Service itself resolved to the dead backend. This is what the change fixes.
.../query_range?query=up{job="pve"} over the last hour → the series flaps between 0 and 1 in long runs (0s until ~1787680840, 1s through ~1787683000, 0s after). Consistent with connection reuse pinning the exporter to one backend at a time, ~1 in 3 of which is dead.
kubernetes/apps/pve/service.yaml has no publishNotReadyAddresses, so ready: false genuinely removes the address from both kube-proxy and Traefik routing.

After Flux reconciles, the reviewer should see up{job="pve"} stop flapping and hold at 1, the pve scrape pool report health: "up" with an empty lastError, pve_node_info / pve_up (currently returning empty vectors because every cluster scrape fails) come back, and pve.renner.uno stop intermittently erroring.

TargetDown{job="pve-node"} will keep firing. That is correct and expected.

Notes

What still needs a human. The alert clears only when jupiter is resolved operationally. Either (a) jupiter comes back and rejoins, at which point flip this endpoint back to ready: true; or (b) jupiter is confirmed permanently retired as a PVE member, at which point it needs a real decommission sweep.
Why I did not do the decommission sweep. jupiter.renner.uno is still wired in as a live PVE host in at least five more places: helmrelease-prometheus.yaml:143 (the pve-node target that is actually alerting), ansible/inventory/pve_hosts.yml:14, ansible/inventory/api.proxmox.yaml:5 and ansible/inventory/group_vars/all/vars.yml:2 (both point the Ansible/OpenTofu Proxmox API at jupiter specifically — these break permanently if jupiter never returns as PVE), and ansible/plays/install-ntp-ha.yml:74 (jupiter is the keepalived NTP MASTER at priority 100). That is a cross-file change well past the surgical bar, it spans ansible/ and opentofu/, and no owner guidance in the issue authorizes it. It should be one deliberate PR, not a side effect of an alert fix.
Direction of travel, for context. ansible/inventory/pve_hosts.yml:74-75 says the NTP role includes edge-1 "so the role survives the PVE hosts being retired for bare-metal Talos"; kubernetes/talos/patches/k8s-1.yaml:1 says k8s-1 runs Talos directly on jupiter's MS-01 hardware; and opentofu/ has vm_151_k8s_2.tf and vm_152_k8s_3.tf but no vm_150_k8s_1.tf. So jupiter most likely is not coming back as a Proxmox node. I could not confirm this from live state — every pve_* metric is currently empty precisely because the cluster scrape fails — so I treated it as strong repo evidence rather than established fact, and wrote the change to be trivially reversible either way.
Unverified, worth a human check: whether jupiter has actually been removed from the Proxmox cluster itself. If it has not, the remaining two-member corosync cluster is one node away from losing quorum. I have no PVE access to check this and it is not repo-fixable.
This is not alert-silencing. No rule, threshold or inhibition was touched, and the alert that prompted the issue still fires. The change restores metric collection that was broken, rather than suppressing a signal.
The issue's recommendation to restore the immich replica counts (kubernetes/apps/immich/server.yaml, machine-learning.yaml) is deliberately left alone — those carry explicit "RESTORE TO 1 ONCE k8s-1 HAS REJOINED" markers and k8s-1 has not rejoined, so restoring them now would schedule workloads onto capacity that does not exist.

Closes #573

## What this PR does **Read this first: this PR does _not_ stop `TargetDown (pve-node)` from firing, and it is not meant to.** The alert in #573 is genuinely not repo-fixable — jupiter is powered down mid-conversion, and no manifest change makes an offline host answer. What I did find, while verifying the issue's claims, is a real and currently-active defect in one of the two files the issue named: `kubernetes/apps/pve/endpointslice.yaml` hardcodes `ready: true` for jupiter's dead `.201` address. Because that EndpointSlice is manually managed, Kubernetes health-checks nothing in it, so kube-proxy and Traefik keep load balancing onto a host that is unreachable. That silently broke the **`pve` cluster scrape job** (a *different* job from the one that alerted) and the `pve.renner.uno` web UI for a large fraction of requests. This PR marks that one endpoint not-ready, which is the supported way to drain a manual endpoint. The issue explicitly asserted "nothing about the scrape config (`helmrelease-prometheus.yaml`, `endpointslice.yaml`) is wrong." That assessment is incorrect, and the evidence is below. ## Changes - `kubernetes/apps/pve/endpointslice.yaml` — set jupiter's endpoint `conditions.ready` to `false` (was `true`) and document why. Removes the dead `10.2.40.201` from the `pve` Service's load-balancing set. The address, hostname and comment history are left intact so restoring it is a one-word change. No other file touched. In particular I did **not** remove `jupiter.renner.uno` from the `pve-node` target list — see Notes. ## Verification Everything below was observed in command output during this run. - `./.ci-diag/kubectl get nodes` → only `k8s-2` and `k8s-3` `Ready`; `k8s-1` absent. Confirms the issue's claim. - `.../query?query=up{job="pve-node"}` → `jupiter.renner.uno = 0`, `neptune = 1`, `saturn = 1`. Confirms the issue's claim. - `.../targets?state=any&scrapePool=pve` → the **cluster** job is also `health: "down"`, `lastError: "server returned HTTP status 500 INTERNAL SERVER ERROR"`. The issue did not notice this. - `./.ci-diag/kubectl logs -n monitoring pve-exporter-b8b97bcd5-rwbtg` → two distinct failures, which is the key evidence: - `HTTPSConnectionPool(host='jupiter.renner.uno', port=8006) ... [Errno 113] Host is unreachable` — the `pve-node` job, expected while jupiter is down. - `HTTPSConnectionPool(host='pve.apps.svc.cluster.local', port=8006) ... [Errno 113] Host is unreachable` — the `pve` **cluster** job, i.e. the Service itself resolved to the dead backend. This is what the change fixes. - `.../query_range?query=up{job="pve"}` over the last hour → the series flaps between `0` and `1` in long runs (0s until ~`1787680840`, 1s through ~`1787683000`, 0s after). Consistent with connection reuse pinning the exporter to one backend at a time, ~1 in 3 of which is dead. - `kubernetes/apps/pve/service.yaml` has no `publishNotReadyAddresses`, so `ready: false` genuinely removes the address from both kube-proxy and Traefik routing. **After Flux reconciles**, the reviewer should see `up{job="pve"}` stop flapping and hold at `1`, the `pve` scrape pool report `health: "up"` with an empty `lastError`, `pve_node_info` / `pve_up` (currently returning empty vectors because every cluster scrape fails) come back, and `pve.renner.uno` stop intermittently erroring. `TargetDown{job="pve-node"}` will keep firing. That is correct and expected. ## Notes - **What still needs a human.** The alert clears only when jupiter is resolved operationally. Either (a) jupiter comes back and rejoins, at which point flip this endpoint back to `ready: true`; or (b) jupiter is confirmed permanently retired as a PVE member, at which point it needs a real decommission sweep. - **Why I did not do the decommission sweep.** `jupiter.renner.uno` is still wired in as a live PVE host in at least five more places: `helmrelease-prometheus.yaml:143` (the `pve-node` target that is actually alerting), `ansible/inventory/pve_hosts.yml:14`, `ansible/inventory/api.proxmox.yaml:5` and `ansible/inventory/group_vars/all/vars.yml:2` (both point the Ansible/OpenTofu Proxmox API at jupiter specifically — these break permanently if jupiter never returns as PVE), and `ansible/plays/install-ntp-ha.yml:74` (jupiter is the keepalived NTP `MASTER` at priority 100). That is a cross-file change well past the surgical bar, it spans `ansible/` and `opentofu/`, and no owner guidance in the issue authorizes it. It should be one deliberate PR, not a side effect of an alert fix. - **Direction of travel, for context.** `ansible/inventory/pve_hosts.yml:74-75` says the NTP role includes edge-1 "so the role survives the PVE hosts being retired for bare-metal Talos"; `kubernetes/talos/patches/k8s-1.yaml:1` says k8s-1 runs Talos directly on jupiter's MS-01 hardware; and `opentofu/` has `vm_151_k8s_2.tf` and `vm_152_k8s_3.tf` but **no** `vm_150_k8s_1.tf`. So jupiter most likely is not coming back as a Proxmox node. I could not confirm this from live state — every `pve_*` metric is currently empty precisely because the cluster scrape fails — so I treated it as strong repo evidence rather than established fact, and wrote the change to be trivially reversible either way. - **Unverified, worth a human check:** whether jupiter has actually been removed from the Proxmox cluster itself. If it has not, the remaining two-member corosync cluster is one node away from losing quorum. I have no PVE access to check this and it is not repo-fixable. - **This is not alert-silencing.** No rule, threshold or inhibition was touched, and the alert that prompted the issue still fires. The change restores metric collection that was broken, rather than suppressing a signal. - The issue's recommendation to restore the immich replica counts (`kubernetes/apps/immich/server.yaml`, `machine-learning.yaml`) is deliberately left alone — those carry explicit "RESTORE TO 1 ONCE k8s-1 HAS REJOINED" markers and k8s-1 has not rejoined, so restoring them now would schedule workloads onto capacity that does not exist. Closes #573

issue-fix-bot added 1 commit 2026-08-25 18:48:50 +00:00

fix: [alert] TargetDown (pve-node) (#573 )

ci/woodpecker/pr/ci-manifests-test Pipeline is pending

Details

ci/woodpecker/pr/ci-scripts-test Pipeline is pending

Details

ci/woodpecker/pr/pr-review Pipeline is pending

Details

a7ccbd7a76

issue-fix-bot referenced this pull request

2026-08-25 18:48:50 +00:00

[alert] TargetDown (pve-node) #573

k.renner closed this pull request

2026-08-26 00:11:52 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#577