fix: [alert] KubeJobFailed (velero) (#1095) #1096

2026-10-01T15:44:34Z

issue-fix-bot commented

2026-10-01 15:44:34 +00:00

What this PR does

KubeJobFailed fired for velero/restore-test-29847360, the monthly restore-verification run on 2026-10-01. The Job hit activeDeadlineSeconds (6 h) while it was still waiting for the immich restore. Each restore had a 4 h phase timeout (RESTORE_PHASE_TIMEOUT_S=14400), but immich only started 2 h 17 m into the run, so that timeout could never fire before the deadline. Kubernetes killed the pod first. Because of that, the summary was never printed, cleanup_service never ran for immich, and the details of the audiobookshelf parity FAIL were lost (they were held only in the in-memory FAILURES array). This PR adds a whole-sweep time budget that is shorter than the Job deadline. Each restore's timeout is capped by the time left in that budget, and once the budget is used up, the remaining services and PVC probes are SKIPped. The script now ends on its own, with a summary and cleanup, before Kubernetes kills it. It also prints each failure's details as soon as the failure happens.

Changes

kubernetes/infrastructure/velero/restore-test.sh
- New SWEEP_BUDGET_S (default 19800 s = 5 h 30 m) and SWEEP_DEADLINE, computed when the script starts.
- Each restore's phase timeout is now min(RESTORE_PHASE_TIMEOUT_S, time left in the budget). The TIMEOUT message shows the timeout that was actually used.
- Once the budget is used up, a service is reported as SKIP (restore (time budget)), and any remaining PVC parity probes for the current service are SKIPped. The summary still runs and lists these, so a backup that was not tested stays visible.
- check() now prints the last 4 output lines of a failed check right after its FAIL line, not only in the final summary. For parity checks those lines include source=N restored=M, so a deadline kill can no longer erase them.
kubernetes/infrastructure/velero/restore-test-cronjob.yaml — sets SWEEP_BUDGET_S: "19800" explicitly, next to the existing timeout, with a comment tying it to activeDeadlineSeconds. The 30 min of slack covers the last service's parity probes (up to about 15 min for wait plus snapshot plus wait), the namespace wait in cleanup_service (up to 5 min), and the summary.

Verification

Checked live during this run:

kubectl -n velero get job restore-test-29847360 -o jsonpath='{.status}' → reason: DeadlineExceeded, startTime 2026-10-01T08:00:00Z, Failed at 14:00:31Z.
Loki {namespace="velero",pod=~"restore-test-29847360.*"} → the last line is [10:17:21] immich: using backup immich-daily-20261001051507. Earlier lines include audiobookshelf: PVC audiobookshelf-books-bulk file count parity (±10% / ±2) FAIL. No capacity ok line appears anywhere, which confirms the run used a script from before the capacity gate.
Prometheus max(ceph_osd_stat_bytes_used/ceph_osd_stat_bytes)*100 → 74.11, which matches the issue.
kubectl -n velero get datadownloads.velero.io -l velero.io/restore-name=audiobookshelf-daily-rtest-1790841606 → No resources found. The books-bulk counts can't be recovered from the cluster anymore.

How to confirm:

After Flux reconciles, kubectl -n velero get cronjob restore-test -o yaml should show the SWEEP_BUDGET_S env var, and the script configmap it mounts should contain SWEEP_DEADLINE.
In the 2026-11-01 run, no Job should reach DeadlineExceeded. A slow sweep should instead end with SKIP ... (time budget) lines and a printed === Summary ===. Any FAIL line should be followed directly by | ... detail lines.
To clear the current alert: kubectl -n velero delete job restore-test-29847360. Otherwise the Job's 40-day TTL keeps it firing until about 2026-11-10. This PR does not do that deletion.

Notes

I could not run bash -n in this environment (blocked by the tool allowlist). The edits were reviewed by hand; please run bash -n kubernetes/infrastructure/velero/restore-test.sh before merging.
The issue's guess that the new capacity gate will SKIP immich next run is not verified. The fullest OSD is at 74.1% now, but whether immich gets skipped depends on its DataUpload sizes on 2026-11-01. This PR makes the sweep end cleanly whether or not the gate skips immich.
Not addressed: the audiobookshelf-books-bulk parity FAIL. The counts were lost and the DataDownloads are gone, so the cause (partial restore vs. library change after the 03:45 backup) is still unknown. Next run, the inline FAIL output will show source=/restored= even if the run is killed. Before that, a human could re-run the sweep for audiobookshelf only to check it. This may be a real backup-integrity problem and needs its own follow-up.
I did not raise activeDeadlineSeconds (the issue's alternative). A longer deadline would only move the same mismatch. The 08:00 + 6 h window was chosen on purpose to stay clear of the next backup window.
Edge case: if the budget runs out between the top-of-loop check and the restore (the capacity gate and pre-cleanup take a few minutes), the restore gets a timeout of 0 or less. It then FAILs immediately with TIMEOUT and is cleaned up. That is noisy, but it is safe and visible.

Closes #1095

## What this PR does `KubeJobFailed` fired for `velero/restore-test-29847360`, the monthly restore-verification run on 2026-10-01. The Job hit `activeDeadlineSeconds` (6 h) while it was still waiting for the immich restore. Each restore had a 4 h phase timeout (`RESTORE_PHASE_TIMEOUT_S=14400`), but immich only started 2 h 17 m into the run, so that timeout could never fire before the deadline. Kubernetes killed the pod first. Because of that, the summary was never printed, `cleanup_service` never ran for immich, and the details of the audiobookshelf parity FAIL were lost (they were held only in the in-memory `FAILURES` array). This PR adds a whole-sweep time budget that is shorter than the Job deadline. Each restore's timeout is capped by the time left in that budget, and once the budget is used up, the remaining services and PVC probes are SKIPped. The script now ends on its own, with a summary and cleanup, before Kubernetes kills it. It also prints each failure's details as soon as the failure happens. ## Changes - `kubernetes/infrastructure/velero/restore-test.sh` - New `SWEEP_BUDGET_S` (default 19800 s = 5 h 30 m) and `SWEEP_DEADLINE`, computed when the script starts. - Each restore's phase timeout is now `min(RESTORE_PHASE_TIMEOUT_S, time left in the budget)`. The `TIMEOUT` message shows the timeout that was actually used. - Once the budget is used up, a service is reported as SKIP (`restore (time budget)`), and any remaining PVC parity probes for the current service are SKIPped. The summary still runs and lists these, so a backup that was not tested stays visible. - `check()` now prints the last 4 output lines of a failed check right after its `FAIL` line, not only in the final summary. For parity checks those lines include `source=N restored=M`, so a deadline kill can no longer erase them. - `kubernetes/infrastructure/velero/restore-test-cronjob.yaml` — sets `SWEEP_BUDGET_S: "19800"` explicitly, next to the existing timeout, with a comment tying it to `activeDeadlineSeconds`. The 30 min of slack covers the last service's parity probes (up to about 15 min for wait plus snapshot plus wait), the namespace wait in `cleanup_service` (up to 5 min), and the summary. ## Verification Checked live during this run: - `kubectl -n velero get job restore-test-29847360 -o jsonpath='{.status}'` → `reason: DeadlineExceeded`, `startTime 2026-10-01T08:00:00Z`, `Failed` at `14:00:31Z`. - Loki `{namespace="velero",pod=~"restore-test-29847360.*"}` → the last line is `[10:17:21] immich: using backup immich-daily-20261001051507`. Earlier lines include `audiobookshelf: PVC audiobookshelf-books-bulk file count parity (±10% / ±2) FAIL`. No `capacity ok` line appears anywhere, which confirms the run used a script from before the capacity gate. - Prometheus `max(ceph_osd_stat_bytes_used/ceph_osd_stat_bytes)*100` → `74.11`, which matches the issue. - `kubectl -n velero get datadownloads.velero.io -l velero.io/restore-name=audiobookshelf-daily-rtest-1790841606` → `No resources found`. The books-bulk counts can't be recovered from the cluster anymore. How to confirm: - After Flux reconciles, `kubectl -n velero get cronjob restore-test -o yaml` should show the `SWEEP_BUDGET_S` env var, and the script configmap it mounts should contain `SWEEP_DEADLINE`. - In the 2026-11-01 run, no Job should reach `DeadlineExceeded`. A slow sweep should instead end with `SKIP ... (time budget)` lines and a printed `=== Summary ===`. Any FAIL line should be followed directly by ` | ...` detail lines. - To clear the current alert: `kubectl -n velero delete job restore-test-29847360`. Otherwise the Job's 40-day TTL keeps it firing until about 2026-11-10. This PR does not do that deletion. ## Notes - I could not run `bash -n` in this environment (blocked by the tool allowlist). The edits were reviewed by hand; please run `bash -n kubernetes/infrastructure/velero/restore-test.sh` before merging. - The issue's guess that the new capacity gate will SKIP immich next run is not verified. The fullest OSD is at 74.1% now, but whether immich gets skipped depends on its DataUpload sizes on 2026-11-01. This PR makes the sweep end cleanly whether or not the gate skips immich. - **Not addressed: the `audiobookshelf-books-bulk` parity FAIL.** The counts were lost and the DataDownloads are gone, so the cause (partial restore vs. library change after the 03:45 backup) is still unknown. Next run, the inline FAIL output will show `source=/restored=` even if the run is killed. Before that, a human could re-run the sweep for audiobookshelf only to check it. This may be a real backup-integrity problem and needs its own follow-up. - I did not raise `activeDeadlineSeconds` (the issue's alternative). A longer deadline would only move the same mismatch. The 08:00 + 6 h window was chosen on purpose to stay clear of the next backup window. - Edge case: if the budget runs out between the top-of-loop check and the restore (the capacity gate and pre-cleanup take a few minutes), the restore gets a timeout of 0 or less. It then FAILs immediately with `TIMEOUT` and is cleaned up. That is noisy, but it is safe and visible. Closes #1095

issue-fix-bot added 1 commit 2026-10-01 15:44:34 +00:00

fix: [alert] KubeJobFailed (velero) (#1095 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline failed

Details

abf987ff98

issue-fix-bot referenced this pull request

2026-10-01 15:44:34 +00:00

[alert] KubeJobFailed (velero) #1095

Ghost closed this pull request

2026-10-02 05:12:25 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1096