VeleroBackupFailed fired because 3 of 12 DataUploads in home-assistant-daily-20260827064538 died when node k8s-1 rebooted 14 minutes into the backup window as part of an operator-driven rolling Talos reboot. That root cause is not fixable in this repo — the reboot is a manual talosctl upgrade, there is no in-cluster reboot scheduler (no kured, no system-upgrade-controller) to coordinate against, and the repo already documents this exact tradeoff as accepted behaviour in prometheusrule-velero.yaml. So this PR does not touch the alert or the schedule times.
What it does fix is a separate, genuine defect surfaced while verifying: the home-assistant-daily schedule was sweeping in ha-pg-1, the live PVC of the ha-pg CNPG Postgres cluster. The schedule's comment asserted "No CNPG cluster in this namespace, so no label exclusion is needed" — that is stale; ha-pg exists, is healthy, and its PVC carries the cnpg.io/cluster label. A crash-consistent snapshot of a running Postgres data directory is not a restorable backup — precisely the trap the wordpress-daily comment in the same file warns about. ha-pg is already covered by Barman. This adds the same cnpg.io/cluster DoesNotExist exclusion the four other CNPG-adjacent schedules in the file already use.
Changes
kubernetes/infrastructure/post-crds/velero-schedules.yaml — add labelSelector: cnpg.io/cluster DoesNotExist to the home-assistant-daily schedule, excluding ha-pg-1 (20 GiB) from the Velero PVC sweep; correct the stale comment that claimed the namespace had no CNPG cluster, and refresh the PVC list (it omitted zigbee2mqtt-data and zwavejs-store, and said "nine PVCs" where there are 12).
Verification
Verified live this run:
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp → the decisive evidence. Backup ran 06:45:38–07:02:46Z. 06:59:23Z k8s-1 NodeNotSchedulable (cordon), 07:01:33Z k8s-1 Rebooted, 07:02:34Z k8s-1 NodeSchedulable. Then k8s-2 07:10:20→07:18:21 and k8s-3 07:20:26→07:23:21. A rolling reboot that began inside the backup window.
kubectl -n velero get datauploads.velero.io -l velero.io/backup-name=home-assistant-daily-20260827064538 → exactly the 3 DataUploads scheduled on k8s-1 are bad (shelly-manager-data Failed, whisper-data Failed, ha-pg-1 Canceled); all 9 on k8s-2/k8s-3 Completed. Message on the Failed ones: exposed snapshot is not ready: no expose result is available for the current node.
kubectl -n velero get pod node-agent-ctplm -o jsonpath=... → lastState.terminated: exitCode: 0, finishedAt: 2026-08-27T07:00:40Z; restarted 07:01:54Z. Consistent with the node reboot, not a crash.
kubectl get nodes -o wide → all three nodes Ready, all on Talos (v1.13.9), none cordoned or tainted. The rolling maintenance is complete as of 07:23Z.
kubectl -n home-assistant get cluster.postgresql.cnpg.io → ha-pg, 1/1 ready, "Cluster in healthy state" — the CNPG cluster the comment denied existed.
kubectl -n home-assistant get pvc -o custom-columns=... → ha-pg-1 carries label cnpg.io/cluster: ha-pg; the other 11 PVCs carry no such label, so they remain included by the new selector.
kubectl -n home-assistant get backups.postgresql.cnpg.io → ha-pg-daily-*completed every day through ha-pg-daily-20260827025000 (today). Barman coverage is current, so excluding ha-pg-1 from Velero removes redundancy, not the only copy.
kubectl create --dry-run=client -f ... → all 12 Schedules parse.
After Flux reconciles, confirm with kubectl -n velero get schedule home-assistant-daily -o jsonpath='{.spec.template.labelSelector}'. Tomorrow's run should produce 11 DataUploads instead of 12, with no ha-pg-1 entry.
VeleroBackupFailed for this schedule should not re-fire once a backup runs without a node reboot overlapping it. This PR does not make it stop firing today — see below.
Notes
Action needed from a human, and it is time-sensitive.velero_backup_last_successful_timestamp{schedule="home-assistant-daily"} = 1787641500 (last Completed run, 2026-08-25). At query time (time() = 1787816645) that is 48.6h stale. VeleroBackupStale (severity: critical) triggers at 60h with for: 10m, i.e. ~19:15Z tonight — before tomorrow's 06:45 run. Now that all three nodes are back and schedulable, running a one-off backup closes both the data gap and the impending critical alert:
(A repo change cannot do this — Velero does not retry failed scheduled backups.)
What the issue got wrong. Worth recording, since it changes the severity picture:
"The two prior days are Completed — this is not chronic, it's an isolated event today." Incorrect. home-assistant-daily-20260826064540 was PartiallyFailed with 5 errors. Today is the second consecutive failure, which is why the 60h stale clock is about to expire. The issue listed 08-25 and 08-24 and skipped 08-26 entirely — on which 11 of 12 schedules cluster-wide failed or partially failed.
"k8s-3 is still on v1.12.11, upgrade in progress." Stale by the time I checked: all three nodes are on v1.13.9 and the rolling reboot finished at 07:23Z.
Root cause framed as a node-agent SandboxChanged / containerd sandbox recreate. That is a downstream symptom; the node-event timeline shows a full cordon-and-reboot of k8s-1.
"3 of the 9 home-assistant PVCs are missing." There are 12 PVCs, and the 3 missing are shelly-manager-data, whisper-data, ha-pg-1.
The issue's suggestion (2) — move the 06:45 schedule out of the maintenance window — I deliberately did not do. Backups occupy 03:15–08:45 by design; there is no reboot automation in-repo whose window could be de-conflicted, and the reboot was ad-hoc. Shuffling a carefully staggered layout against an arbitrary manual operation would not prevent recurrence.
Alert was not silenced.kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml is untouched. Its VeleroBackupStale comment (raised 30h→60h in #658, the most recent commit) already states the owner's position: "draining a node kills whatever upload is in flight and the schedule is simply lost for that day … the per-event VeleroBackupFailed alert fires on the first failure regardless — so nothing is hidden." By that rationale this alert firing is working as designed, and the ha-pg-1 exclusion is the only defensible repo change here.
Scope caveat. The ha-pg-1 exclusion is a correctness fix, not a fix for the alert's proximate cause. It does shrink the failure surface by one volume, and ha-pg-1 was one of the three that failed — but had the reboot not happened, the backup would have succeeded with ha-pg-1 included and would still have been storing an unrestorable database snapshot. If you would rather keep this PR strictly to the alert, dropping the change entirely is reasonable; the manual backup command above is the part that actually matters today.
## What this PR does
`VeleroBackupFailed` fired because 3 of 12 DataUploads in `home-assistant-daily-20260827064538` died when **node `k8s-1` rebooted 14 minutes into the backup window** as part of an operator-driven rolling Talos reboot. That root cause is **not fixable in this repo** — the reboot is a manual `talosctl upgrade`, there is no in-cluster reboot scheduler (no kured, no system-upgrade-controller) to coordinate against, and the repo already documents this exact tradeoff as accepted behaviour in `prometheusrule-velero.yaml`. So this PR does **not** touch the alert or the schedule times.
What it does fix is a separate, genuine defect surfaced while verifying: the `home-assistant-daily` schedule was sweeping in `ha-pg-1`, the live PVC of the `ha-pg` CNPG Postgres cluster. The schedule's comment asserted "No CNPG cluster in this namespace, so no label exclusion is needed" — that is stale; `ha-pg` exists, is healthy, and its PVC carries the `cnpg.io/cluster` label. A crash-consistent snapshot of a running Postgres data directory is not a restorable backup — precisely the trap the `wordpress-daily` comment in the same file warns about. `ha-pg` is already covered by Barman. This adds the same `cnpg.io/cluster DoesNotExist` exclusion the four other CNPG-adjacent schedules in the file already use.
## Changes
- `kubernetes/infrastructure/post-crds/velero-schedules.yaml` — add `labelSelector: cnpg.io/cluster DoesNotExist` to the `home-assistant-daily` schedule, excluding `ha-pg-1` (20 GiB) from the Velero PVC sweep; correct the stale comment that claimed the namespace had no CNPG cluster, and refresh the PVC list (it omitted `zigbee2mqtt-data` and `zwavejs-store`, and said "nine PVCs" where there are 12).
## Verification
Verified live this run:
- `kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp` → the decisive evidence. Backup ran 06:45:38–07:02:46Z. `06:59:23Z k8s-1 NodeNotSchedulable` (cordon), `07:01:33Z k8s-1 Rebooted`, `07:02:34Z k8s-1 NodeSchedulable`. Then `k8s-2` 07:10:20→07:18:21 and `k8s-3` 07:20:26→07:23:21. A rolling reboot that began *inside* the backup window.
- `kubectl -n velero get datauploads.velero.io -l velero.io/backup-name=home-assistant-daily-20260827064538` → exactly the 3 DataUploads scheduled on `k8s-1` are bad (`shelly-manager-data` Failed, `whisper-data` Failed, `ha-pg-1` Canceled); all 9 on `k8s-2`/`k8s-3` Completed. Message on the Failed ones: `exposed snapshot is not ready: no expose result is available for the current node`.
- `kubectl -n velero get pod node-agent-ctplm -o jsonpath=...` → `lastState.terminated`: `exitCode: 0`, `finishedAt: 2026-08-27T07:00:40Z`; restarted `07:01:54Z`. Consistent with the node reboot, not a crash.
- `kubectl get nodes -o wide` → all three nodes `Ready`, all on `Talos (v1.13.9)`, none cordoned or tainted. **The rolling maintenance is complete as of 07:23Z.**
- `kubectl -n home-assistant get cluster.postgresql.cnpg.io` → `ha-pg`, 1/1 ready, "Cluster in healthy state" — the CNPG cluster the comment denied existed.
- `kubectl -n home-assistant get pvc -o custom-columns=...` → `ha-pg-1` carries label `cnpg.io/cluster: ha-pg`; the other 11 PVCs carry no such label, so they remain included by the new selector.
- `kubectl -n home-assistant get backups.postgresql.cnpg.io` → `ha-pg-daily-*` `completed` every day through `ha-pg-daily-20260827025000` (today). Barman coverage is current, so excluding `ha-pg-1` from Velero removes redundancy, not the only copy.
- `kubectl create --dry-run=client -f ...` → all 12 Schedules parse.
After Flux reconciles, confirm with `kubectl -n velero get schedule home-assistant-daily -o jsonpath='{.spec.template.labelSelector}'`. Tomorrow's run should produce **11** DataUploads instead of 12, with no `ha-pg-1` entry.
`VeleroBackupFailed` for this schedule should not re-fire once a backup runs without a node reboot overlapping it. This PR does not make it stop firing today — see below.
## Notes
**Action needed from a human, and it is time-sensitive.** `velero_backup_last_successful_timestamp{schedule="home-assistant-daily"}` = `1787641500` (last *Completed* run, 2026-08-25). At query time (`time()` = `1787816645`) that is **48.6h** stale. `VeleroBackupStale` (severity: **critical**) triggers at 60h with `for: 10m`, i.e. **~19:15Z tonight** — before tomorrow's 06:45 run. Now that all three nodes are back and schedulable, running a one-off backup closes both the data gap and the impending critical alert:
```
kubectl -n velero create backup home-assistant-manual-20260827 --from-schedule home-assistant-daily
```
(A repo change cannot do this — Velero does not retry failed scheduled backups.)
**What the issue got wrong.** Worth recording, since it changes the severity picture:
1. *"The two prior days are `Completed` — this is not chronic, it's an isolated event today."* Incorrect. `home-assistant-daily-20260826064540` was **`PartiallyFailed` with 5 errors**. Today is the **second consecutive failure**, which is why the 60h stale clock is about to expire. The issue listed 08-25 and 08-24 and skipped 08-26 entirely — on which **11 of 12 schedules cluster-wide failed or partially failed**.
2. *"`k8s-3` is still on `v1.12.11`, upgrade in progress."* Stale by the time I checked: all three nodes are on `v1.13.9` and the rolling reboot finished at 07:23Z.
3. *Root cause framed as a node-agent `SandboxChanged` / containerd sandbox recreate.* That is a downstream symptom; the node-event timeline shows a full cordon-and-reboot of `k8s-1`.
4. *"3 of the 9 home-assistant PVCs are missing."* There are 12 PVCs, and the 3 missing are `shelly-manager-data`, `whisper-data`, `ha-pg-1`.
5. The issue's suggestion (2) — move the 06:45 schedule out of the maintenance window — I deliberately did **not** do. Backups occupy 03:15–08:45 by design; there is no reboot automation in-repo whose window could be de-conflicted, and the reboot was ad-hoc. Shuffling a carefully staggered layout against an arbitrary manual operation would not prevent recurrence.
**Alert was not silenced.** `kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml` is untouched. Its `VeleroBackupStale` comment (raised 30h→60h in #658, the most recent commit) already states the owner's position: "draining a node kills whatever upload is in flight and the schedule is simply lost for that day … the per-event `VeleroBackupFailed` alert fires on the first failure regardless — so nothing is hidden." By that rationale this alert firing is working as designed, and the `ha-pg-1` exclusion is the only defensible repo change here.
**Scope caveat.** The `ha-pg-1` exclusion is a correctness fix, not a fix for the alert's proximate cause. It does shrink the failure surface by one volume, and `ha-pg-1` was one of the three that failed — but had the reboot not happened, the backup would have succeeded with `ha-pg-1` included and would still have been storing an unrestorable database snapshot. If you would rather keep this PR strictly to the alert, dropping the change entirely is reasonable; the manual backup command above is the part that actually matters today.
Closes #657
Nachgeprüft, hält alles — und der Teil unter „Action needed" ist inzwischen erledigt.
Unabhängig bestätigt:
ha-pg existiert in home-assistant, 1/1 ready, Alter 7d20h. Das erklärt den veralteten Kommentar: er war beim Schreiben richtig und ist vor gut einer Woche unwahr geworden, als der HA-Recorder auf Postgres umzog.
ha-pg-1 ist die einzige der zwölf PVCs mit cnpg.io/cluster (Wert ha-pg); die übrigen elf tragen kein Label und bleiben vom neuen Selector eingeschlossen.
Barman ist aktuell: ha-pg-daily-* täglich completed bis ha-pg-daily-20260827025000 (heute, 03:43:26Z). kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml existiert mit schedule: "0 50 2 * * *" — die im Kommentar genannten 02:50 UTC stimmen.
Das Hausmuster stimmt ebenfalls: dieselbe cnpg.io/cluster / DoesNotExist-Ausnahme steht in derselben Datei schon in den Zeilen 48, 96, 143 und 313.
Punkt 1 der Korrekturliste ist richtig und wichtiger als es klingt.home-assistant-daily-20260826064540 ist PartiallyFailed mit 5 Fehlern — heute war der zweite Ausfall in Folge. Davor 30 aufeinanderfolgende Läufe Completed, vom 29.07. bis 25.08. Beide Ausfälle lagen in Node-Wartungsfenstern. Das Muster ist also nicht „Velero ist kaputt", sondern „Backup-Fenster und Node-Wartung überlappen".
Das einmalige Backup ist schon gelaufen.home-assistant-daily-20260827-gapfill, 09:55:40–10:12:48Z, Completed, 12/12 DataUploads — einschließlich shelly-manager-data, whisper-data und ha-pg-1. Die 60h-Uhr für VeleroBackupStale ist damit zurückgesetzt, heute Abend um ~19:15Z passiert nichts mehr.
Eine Korrektur am vorgeschlagenen Befehl: kubectl -n velero create backup ... --from-schedule ... gibt es nicht. --from-schedule ist ein Flag der Velero-CLI, die hier nicht installiert ist, und create backup ist kein kubectl-Verb. Ich habe stattdessen ein Backup-CR aus dem Schedule-Template angelegt.
Zum Scope-Vorbehalt: ich würde die Änderung drin lassen. Sie steht für sich — ha-pg-1 gehörte unabhängig vom Reboot nicht in den Velero-Sweep, und der PR sagt selbst korrekt, dass das die proximate Ursache nicht behebt. Das ist die ehrliche Variante.
Closes #657 läuft ins Leere, das Issue habe ich vorhin schon geschlossen. Nach dem Merge sollten es morgen 11 DataUploads statt 12 sein.
Nachgeprüft, hält alles — und der Teil unter „Action needed" ist inzwischen erledigt.
**Unabhängig bestätigt:**
- `ha-pg` existiert in `home-assistant`, 1/1 ready, Alter **7d20h**. Das erklärt den veralteten Kommentar: er war beim Schreiben richtig und ist vor gut einer Woche unwahr geworden, als der HA-Recorder auf Postgres umzog.
- `ha-pg-1` ist die einzige der zwölf PVCs mit `cnpg.io/cluster` (Wert `ha-pg`); die übrigen elf tragen kein Label und bleiben vom neuen Selector eingeschlossen.
- Barman ist aktuell: `ha-pg-daily-*` täglich `completed` bis `ha-pg-daily-20260827025000` (heute, 03:43:26Z). `kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml` existiert mit `schedule: "0 50 2 * * *"` — die im Kommentar genannten 02:50 UTC stimmen.
- Das Hausmuster stimmt ebenfalls: dieselbe `cnpg.io/cluster` / `DoesNotExist`-Ausnahme steht in derselben Datei schon in den Zeilen 48, 96, 143 und 313.
**Punkt 1 der Korrekturliste ist richtig und wichtiger als es klingt.** `home-assistant-daily-20260826064540` ist `PartiallyFailed` mit 5 Fehlern — heute war der zweite Ausfall in Folge. Davor **30 aufeinanderfolgende Läufe `Completed`**, vom 29.07. bis 25.08. Beide Ausfälle lagen in Node-Wartungsfenstern. Das Muster ist also nicht „Velero ist kaputt", sondern „Backup-Fenster und Node-Wartung überlappen".
**Das einmalige Backup ist schon gelaufen.** `home-assistant-daily-20260827-gapfill`, 09:55:40–10:12:48Z, `Completed`, **12/12 DataUploads** — einschließlich `shelly-manager-data`, `whisper-data` und `ha-pg-1`. Die 60h-Uhr für `VeleroBackupStale` ist damit zurückgesetzt, heute Abend um ~19:15Z passiert nichts mehr.
Eine Korrektur am vorgeschlagenen Befehl: `kubectl -n velero create backup ... --from-schedule ...` gibt es nicht. `--from-schedule` ist ein Flag der Velero-CLI, die hier nicht installiert ist, und `create backup` ist kein kubectl-Verb. Ich habe stattdessen ein Backup-CR aus dem Schedule-Template angelegt.
**Zum Scope-Vorbehalt:** ich würde die Änderung drin lassen. Sie steht für sich — `ha-pg-1` gehörte unabhängig vom Reboot nicht in den Velero-Sweep, und der PR sagt selbst korrekt, dass das die proximate Ursache nicht behebt. Das ist die ehrliche Variante.
`Closes #657` läuft ins Leere, das Issue habe ich vorhin schon geschlossen. Nach dem Merge sollten es morgen 11 DataUploads statt 12 sein.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What this PR does
VeleroBackupFailedfired because 3 of 12 DataUploads inhome-assistant-daily-20260827064538died when nodek8s-1rebooted 14 minutes into the backup window as part of an operator-driven rolling Talos reboot. That root cause is not fixable in this repo — the reboot is a manualtalosctl upgrade, there is no in-cluster reboot scheduler (no kured, no system-upgrade-controller) to coordinate against, and the repo already documents this exact tradeoff as accepted behaviour inprometheusrule-velero.yaml. So this PR does not touch the alert or the schedule times.What it does fix is a separate, genuine defect surfaced while verifying: the
home-assistant-dailyschedule was sweeping inha-pg-1, the live PVC of theha-pgCNPG Postgres cluster. The schedule's comment asserted "No CNPG cluster in this namespace, so no label exclusion is needed" — that is stale;ha-pgexists, is healthy, and its PVC carries thecnpg.io/clusterlabel. A crash-consistent snapshot of a running Postgres data directory is not a restorable backup — precisely the trap thewordpress-dailycomment in the same file warns about.ha-pgis already covered by Barman. This adds the samecnpg.io/cluster DoesNotExistexclusion the four other CNPG-adjacent schedules in the file already use.Changes
kubernetes/infrastructure/post-crds/velero-schedules.yaml— addlabelSelector: cnpg.io/cluster DoesNotExistto thehome-assistant-dailyschedule, excludingha-pg-1(20 GiB) from the Velero PVC sweep; correct the stale comment that claimed the namespace had no CNPG cluster, and refresh the PVC list (it omittedzigbee2mqtt-dataandzwavejs-store, and said "nine PVCs" where there are 12).Verification
Verified live this run:
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp→ the decisive evidence. Backup ran 06:45:38–07:02:46Z.06:59:23Z k8s-1 NodeNotSchedulable(cordon),07:01:33Z k8s-1 Rebooted,07:02:34Z k8s-1 NodeSchedulable. Thenk8s-207:10:20→07:18:21 andk8s-307:20:26→07:23:21. A rolling reboot that began inside the backup window.kubectl -n velero get datauploads.velero.io -l velero.io/backup-name=home-assistant-daily-20260827064538→ exactly the 3 DataUploads scheduled onk8s-1are bad (shelly-manager-dataFailed,whisper-dataFailed,ha-pg-1Canceled); all 9 onk8s-2/k8s-3Completed. Message on the Failed ones:exposed snapshot is not ready: no expose result is available for the current node.kubectl -n velero get pod node-agent-ctplm -o jsonpath=...→lastState.terminated:exitCode: 0,finishedAt: 2026-08-27T07:00:40Z; restarted07:01:54Z. Consistent with the node reboot, not a crash.kubectl get nodes -o wide→ all three nodesReady, all onTalos (v1.13.9), none cordoned or tainted. The rolling maintenance is complete as of 07:23Z.kubectl -n home-assistant get cluster.postgresql.cnpg.io→ha-pg, 1/1 ready, "Cluster in healthy state" — the CNPG cluster the comment denied existed.kubectl -n home-assistant get pvc -o custom-columns=...→ha-pg-1carries labelcnpg.io/cluster: ha-pg; the other 11 PVCs carry no such label, so they remain included by the new selector.kubectl -n home-assistant get backups.postgresql.cnpg.io→ha-pg-daily-*completedevery day throughha-pg-daily-20260827025000(today). Barman coverage is current, so excludingha-pg-1from Velero removes redundancy, not the only copy.kubectl create --dry-run=client -f ...→ all 12 Schedules parse.After Flux reconciles, confirm with
kubectl -n velero get schedule home-assistant-daily -o jsonpath='{.spec.template.labelSelector}'. Tomorrow's run should produce 11 DataUploads instead of 12, with noha-pg-1entry.VeleroBackupFailedfor this schedule should not re-fire once a backup runs without a node reboot overlapping it. This PR does not make it stop firing today — see below.Notes
Action needed from a human, and it is time-sensitive.
velero_backup_last_successful_timestamp{schedule="home-assistant-daily"}=1787641500(last Completed run, 2026-08-25). At query time (time()=1787816645) that is 48.6h stale.VeleroBackupStale(severity: critical) triggers at 60h withfor: 10m, i.e. ~19:15Z tonight — before tomorrow's 06:45 run. Now that all three nodes are back and schedulable, running a one-off backup closes both the data gap and the impending critical alert:(A repo change cannot do this — Velero does not retry failed scheduled backups.)
What the issue got wrong. Worth recording, since it changes the severity picture:
Completed— this is not chronic, it's an isolated event today." Incorrect.home-assistant-daily-20260826064540wasPartiallyFailedwith 5 errors. Today is the second consecutive failure, which is why the 60h stale clock is about to expire. The issue listed 08-25 and 08-24 and skipped 08-26 entirely — on which 11 of 12 schedules cluster-wide failed or partially failed.k8s-3is still onv1.12.11, upgrade in progress." Stale by the time I checked: all three nodes are onv1.13.9and the rolling reboot finished at 07:23Z.SandboxChanged/ containerd sandbox recreate. That is a downstream symptom; the node-event timeline shows a full cordon-and-reboot ofk8s-1.shelly-manager-data,whisper-data,ha-pg-1.Alert was not silenced.
kubernetes/apps/monitoring-rules/prometheusrule-velero.yamlis untouched. ItsVeleroBackupStalecomment (raised 30h→60h in #658, the most recent commit) already states the owner's position: "draining a node kills whatever upload is in flight and the schedule is simply lost for that day … the per-eventVeleroBackupFailedalert fires on the first failure regardless — so nothing is hidden." By that rationale this alert firing is working as designed, and theha-pg-1exclusion is the only defensible repo change here.Scope caveat. The
ha-pg-1exclusion is a correctness fix, not a fix for the alert's proximate cause. It does shrink the failure surface by one volume, andha-pg-1was one of the three that failed — but had the reboot not happened, the backup would have succeeded withha-pg-1included and would still have been storing an unrestorable database snapshot. If you would rather keep this PR strictly to the alert, dropping the change entirely is reasonable; the manual backup command above is the part that actually matters today.Closes #657
Nachgeprüft, hält alles — und der Teil unter „Action needed" ist inzwischen erledigt.
Unabhängig bestätigt:
ha-pgexistiert inhome-assistant, 1/1 ready, Alter 7d20h. Das erklärt den veralteten Kommentar: er war beim Schreiben richtig und ist vor gut einer Woche unwahr geworden, als der HA-Recorder auf Postgres umzog.ha-pg-1ist die einzige der zwölf PVCs mitcnpg.io/cluster(Wertha-pg); die übrigen elf tragen kein Label und bleiben vom neuen Selector eingeschlossen.ha-pg-daily-*täglichcompletedbisha-pg-daily-20260827025000(heute, 03:43:26Z).kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yamlexistiert mitschedule: "0 50 2 * * *"— die im Kommentar genannten 02:50 UTC stimmen.cnpg.io/cluster/DoesNotExist-Ausnahme steht in derselben Datei schon in den Zeilen 48, 96, 143 und 313.Punkt 1 der Korrekturliste ist richtig und wichtiger als es klingt.
home-assistant-daily-20260826064540istPartiallyFailedmit 5 Fehlern — heute war der zweite Ausfall in Folge. Davor 30 aufeinanderfolgende LäufeCompleted, vom 29.07. bis 25.08. Beide Ausfälle lagen in Node-Wartungsfenstern. Das Muster ist also nicht „Velero ist kaputt", sondern „Backup-Fenster und Node-Wartung überlappen".Das einmalige Backup ist schon gelaufen.
home-assistant-daily-20260827-gapfill, 09:55:40–10:12:48Z,Completed, 12/12 DataUploads — einschließlichshelly-manager-data,whisper-dataundha-pg-1. Die 60h-Uhr fürVeleroBackupStaleist damit zurückgesetzt, heute Abend um ~19:15Z passiert nichts mehr.Eine Korrektur am vorgeschlagenen Befehl:
kubectl -n velero create backup ... --from-schedule ...gibt es nicht.--from-scheduleist ein Flag der Velero-CLI, die hier nicht installiert ist, undcreate backupist kein kubectl-Verb. Ich habe stattdessen ein Backup-CR aus dem Schedule-Template angelegt.Zum Scope-Vorbehalt: ich würde die Änderung drin lassen. Sie steht für sich —
ha-pg-1gehörte unabhängig vom Reboot nicht in den Velero-Sweep, und der PR sagt selbst korrekt, dass das die proximate Ursache nicht behebt. Das ist die ehrliche Variante.Closes #657läuft ins Leere, das Issue habe ich vorhin schon geschlossen. Nach dem Merge sollten es morgen 11 DataUploads statt 12 sein.