fix: [alert] KubeJobFailed (velero) (#1095 )

Merge pull request 'zigbee2mqtt: apply WS90 reporting config via external converter' (#1092 ) from z2m-ws90-converter into master
Reviewed-on: k.renner/homelab#1092
2026-10-01 15:44:33 +00:00 · 2026-10-01 10:40:53 +00:00 · 2026-10-01 10:31:00 +00:00 · 2026-10-01 10:25:09 +00:00 · 2026-10-01 12:01:21 +02:00 · 2026-10-01 09:53:24 +00:00
33 changed files with 1157 additions and 64 deletions
@@ -0,0 +1,69 @@
+# Paperclip server image (REN-401): upstream paperclipai/paperclip at a pinned
+# master commit plus the patches in ci/images/paperclip/. Needed while the k8s
+# sandbox fix (upstream PR #13733) is not merged; see the README there for how
+# to go back to upstream nightly.
+#
+# Own workflow for the reason given in build-agent-runtime.yml: a step in
+# build-ci-images.yml runs on every change to that file and competes for the
+# woodpecker-ci-quota. Re-run alone with a deployment to target
+# `build-paperclip`.
+#
+# Tag `build.<unix-ts>` is what the Flux ImagePolicy `paperclip` sorts on
+# (image-automation/paperclip.yaml); image automation then rolls it out.
+# konairius/ and ci_images_token as for the agent-runtime images.
+when:
+  - event: push
+    branch: master
+    path:
+      - "ci/images/paperclip/**"
+      - ".woodpecker/build-paperclip.yml"
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-paperclip"'
+
+steps:
+  - name: prepare-source
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
+    commands:
+      - sh ci/images/paperclip/prepare.sh build/paperclip-src
+
+  - name: build-paperclip
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: build/paperclip-src
+    depends_on: [prepare-source]
+    # Rust runner + UI + server build in one pod. The server build alone sets
+    # a 4 GiB Node heap, so the memory limit is the LimitRange max (8Gi).
+    backend_options:
+      kubernetes:
+        resources:
+          requests:
+            cpu: "1"
+            memory: 2Gi
+          limits:
+            cpu: "4"
+            memory: 8Gi
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - . ../../ci/images/paperclip/upstream.env
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=build.$(date +%s)
+      # Same target and build args as upstream's docker.yml for nightly.
+      # --compressed-caching=false: pipeline 11717 was killed at the 8Gi limit
+      # (exit 137) in the server build; kaniko otherwise holds compressed
+      # layer tarballs in memory next to the 4 GiB Node heap.
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Dockerfile.kaniko \
+          --target production \
+          --skip-unused-stages \
+          --snapshot-mode=redo \
+          --compressed-caching=false \
+          --build-arg PAPERCLIP_BUILD_COMMIT=$PAPERCLIP_COMMIT \
+          --build-arg PAPERCLIP_BUILD_VERSION=$PAPERCLIP_VERSION \
+          --build-arg CLI_TOOLS_CACHE_EPOCH=$(date -u +%G-W%V) \
+          --destination gitea.renner.uno/konairius/paperclip:$TAG
@@ -0,0 +1,55 @@
+# Paperclip server image (REN-401)
+
+Upstream `paperclipai/paperclip` at the commit in `upstream.env`, plus every
+`*.patch` in this directory. Built by `.woodpecker/build-paperclip.yml`,
+pushed as `gitea.renner.uno/konairius/paperclip:build.<unix-ts>`, rolled out
+by Flux image automation (`kubernetes/infrastructure/image-automation/paperclip.yaml`).
+
+## Why
+
+k8s sandbox runs fail: the run path never sets the lease `remoteCwd`, so file
+sync into the pod fails (paperclipai/paperclip#13864). The fix is
+paperclipai/paperclip#13733, open and not merged.
+`pr-13733-lease-remote-cwd.patch` is that PR's diff (head `5f320bf6`).
+
+#13733 covers only the run path. Adapter environment tests (binding an agent
+to a k8s environment) sync managed config before any workspace is realized,
+so the lease still has no `remoteCwd` and the Claude test fails with
+`claude_managed_config_dir_failed` (REN-409). The fix is
+paperclipai/paperclip#14769, also open: the plugin sets `remoteCwd:
+/workspace` when it acquires or resumes a lease.
+`pr-14769-probe-remote-cwd.patch` is that PR's diff (head `10dc7371`).
+
+## Files
+
+- `upstream.env` — pinned commit and the version string stamped into the
+  image. Bump both together.
+- `prepare.sh` — fetches the commit, applies the patches, writes
+  `Dockerfile.kaniko` (upstream's `COPY --parents` line split into plain
+  `COPY` lines; kaniko does not support `--parents`), and adds the build of
+  the Kubernetes sandbox provider plugin to the `build` stage (REN-405):
+  upstream's `production` target ships only its source, so the plugin fails
+  with "no longer exposes a Paperclip manifest". The build fails if
+  `dist/manifest.js` is missing.
+- `*.patch` — applied in name order with `git apply`.
+
+## Moving to a newer upstream commit
+
+Set `PAPERCLIP_COMMIT` to the new master commit and `PAPERCLIP_VERSION` to
+`git describe --tags --match 'v*' --long <commit>`, then run
+`sh prepare.sh /tmp/src`: it fails if a patch no longer applies.
+
+## Exit: when #13733 and #14769 are merged upstream
+
+Renovate cannot watch an upstream PR, so this is a manual step. Once
+upstream nightly contains both fixes:
+
+1. `kubernetes/infrastructure/image-automation/paperclip.yaml`: back to
+   `ghcr.io/paperclipai/paperclip` with `filterTags.pattern: '^nightly$'`
+   (the version before REN-401 in git history).
+2. `renovate.json`: swap `gitea.renner.uno/konairius/paperclip` back to
+   `ghcr.io/paperclipai/paperclip` in the image-automation exclusion list.
+3. Delete this directory and `.woodpecker/build-paperclip.yml`.
+
+Image automation then rewrites `kubernetes/apps/paperclip/paperclip.yaml`
+to the nightly digest by itself.
@@ -0,0 +1,93 @@
+diff --git a/server/src/__tests__/environment-run-orchestrator.test.ts b/server/src/__tests__/environment-run-orchestrator.test.ts
+index f3d1fc659..812cae357 100644
+--- a/server/src/__tests__/environment-run-orchestrator.test.ts
+++ b/server/src/__tests__/environment-run-orchestrator.test.ts
+@@ -430,6 +430,51 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
+     expect(result.persistedExecutionWorkspace).toEqual(updatedEw);
+   });
+ 
+  it("persists the realized cwd as remoteCwd when realization returns no workspaceRealization payload", async () => {
+    const runtime = makeMockRuntime({
+      realizeWorkspace: vi.fn().mockResolvedValue({
+        cwd: "/workspace",
+        metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
+      }),
+    } as Partial<EnvironmentRuntimeService>);
+    mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
+
+    const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
+
+    await orchestrator.realizeForRun(
+      makeRealizeInput({ environment: makeEnvironment("sandbox") }),
+    );
+
+    expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(
+      "lease-1",
+      expect.objectContaining({ remoteCwd: "/workspace" }),
+    );
+    // Nothing to record about the realization shape, so the execution workspace
+    // row is left untouched.
+    expect(mockUpdateExecutionWorkspace).not.toHaveBeenCalled();
+  });
+
+  it("does not rewrite lease metadata when the realized cwd already matches the lease", async () => {
+    const runtime = makeMockRuntime({
+      realizeWorkspace: vi.fn().mockResolvedValue({
+        cwd: "/workspace",
+        metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
+      }),
+    } as Partial<EnvironmentRuntimeService>);
+    mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
+
+    const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
+
+    await orchestrator.realizeForRun(
+      makeRealizeInput({
+        environment: makeEnvironment("sandbox"),
+        lease: makeLease({ metadata: { remoteCwd: "/workspace" } }),
+      }),
+    );
+
+    expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
+  });
+
+   it("runs a remote provision command after workspace realization when configured", async () => {
+     mockBuildWorkspaceRealizationRequest.mockReturnValue({
+       version: 1,
+diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts
+index 97cca667e..e76f64913 100644
+--- a/server/src/services/environment-run-orchestrator.ts
+++ b/server/src/services/environment-run-orchestrator.ts
+@@ -485,16 +485,29 @@ export function environmentRunOrchestrator(
+     }
+ 
+     // Step 3: Persist realization metadata on lease and execution workspace
+-    if (Object.keys(workspaceRealization).length > 0) {
+    const hasWorkspaceRealization = Object.keys(workspaceRealization).length > 0;
+    // A driver that realizes the workspace reports the directory it prepared
+    // through the realization `cwd`. Persist it on the lease as `remoteCwd`, so
+    // every later step that receives only the lease — native file sync, a
+    // resumed lease, a second run against the same lease — confines itself to
+    // the same root. Without it a driver whose realization returns no
+    // `workspaceRealization` payload leaves the lease without a root at all,
+    // and the sync step fails even though realization succeeded.
+    const currentRemoteCwd =
+      typeof lease.metadata?.remoteCwd === "string" ? lease.metadata.remoteCwd.trim() : "";
+    const realizedRemoteCwd =
+      realizedWorkspaceCwd && realizedWorkspaceCwd !== currentRemoteCwd ? realizedWorkspaceCwd : null;
+    if (hasWorkspaceRealization || realizedRemoteCwd) {
+       const nextLeaseMetadata = {
+         ...(lease.metadata ?? {}),
+-        workspaceRealization,
+        ...(hasWorkspaceRealization ? { workspaceRealization } : {}),
+        ...(realizedRemoteCwd ? { remoteCwd: realizedRemoteCwd } : {}),
+       };
+       const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, nextLeaseMetadata);
+       if (updatedLease) {
+         lease = updatedLease;
+       }
+-      if (persistedExecutionWorkspace) {
+      if (hasWorkspaceRealization && persistedExecutionWorkspace) {
+         const updatedEw = await executionWorkspacesSvc.update(persistedExecutionWorkspace.id, {
+           metadata: {
+             ...(persistedExecutionWorkspace.metadata ?? {}),
@@ -0,0 +1,86 @@
+diff --git a/packages/plugins/sandbox-providers/kubernetes/README.md b/packages/plugins/sandbox-providers/kubernetes/README.md
+index a51c6ecee60e..72618cb11296 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/README.md
+++ b/packages/plugins/sandbox-providers/kubernetes/README.md
+@@ -140,6 +140,8 @@ Every agent pod is:
+ 
+ Plus per-namespace `pod-security.kubernetes.io/enforce: restricted` and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, configured FQDNs/CIDRs).
+ 
+The lease starts with `/workspace` as the native file-sync confinement root. Adapter environment tests can upload managed config before the server realizes a workspace. When resuming a lease, the provider keeps a previously recorded `remoteCwd` so a custom realized workspace is not replaced with the default.
+
+ The per-run Secret carrying the bootstrap token and adapter API keys has `ownerReferences` pointing at the owning Job, so a single `kubectl delete job …` cascades cleanly to the Pod and Secret.
+ 
+ ## Optional Kata-FC microVM isolation
+diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+index 8ae47de8ecac..98016385da05 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+@@ -454,6 +454,9 @@ const plugin = definePlugin({
+       secretName,
+       phase: "Pending",
+       backend: config.backend,
+      // Adapter environment probes sync managed assets before workspace
+      // realization. Give native sync the pod's mounted confinement root now.
+      remoteCwd: "/workspace",
+       scopedNetworkPolicyName,
+       scopedNetworkEgress,
+       // Native file sync streams over a pod exec; only the sandbox-cr backend
+@@ -529,6 +532,10 @@ const plugin = definePlugin({
+       secretName,
+       phase: check.phase,
+       backend: leaseBackend,
+      remoteCwd:
+        typeof params.leaseMetadata?.remoteCwd === "string" && params.leaseMetadata.remoteCwd.trim()
+          ? params.leaseMetadata.remoteCwd.trim()
+          : "/workspace",
+       scopedNetworkPolicyName:
+         typeof params.leaseMetadata?.scopedNetworkPolicyName === "string"
+           ? params.leaseMetadata.scopedNetworkPolicyName
+diff --git a/packages/plugins/sandbox-providers/kubernetes/src/types.ts b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+index 5626daa93846..ef3d0d889a4c 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+@@ -90,6 +90,8 @@ export interface KubernetesLeaseMetadata {
+   phase: "Pending" | "Running" | "Succeeded" | "Failed";
+   /** Which backend provisioned this lease. */
+   backend: "sandbox-cr" | "job";
+  /** Native sync confinement root; present before workspace realization for probes. */
+  remoteCwd: string;
+   scopedNetworkPolicyName: string | null;
+   scopedNetworkEgress: {
+     allowFqdns: string[];
+diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+index 7ad0ca4807e5..4c23410d4d15 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+@@ -80,6 +80,7 @@ describe("onEnvironmentResumeLease", () => {
+         secretName: "pc-abc-env",
+         phase: "Running",
+         backend: "sandbox-cr",
+        remoteCwd: "/workspace",
+         resumedLease: true,
+         // sandbox-cr has a pod-exec channel, so native file sync stays enabled.
+         nativeFileSyncUnsupported: false,
+@@ -87,6 +88,22 @@ describe("onEnvironmentResumeLease", () => {
+     );
+   });
+ 
+  it("keeps a custom realized cwd when resuming a sandbox lease", async () => {
+    h.clients = {
+      custom: { getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr("pc-abc-pod")) },
+      core: { readNamespacedPod: vi.fn().mockResolvedValue({ status: { phase: "Running" } }) },
+    };
+    const lease = await plugin.definition.onEnvironmentResumeLease!({
+      driverKey: "kubernetes",
+      companyId: "acme",
+      environmentId: "env-1",
+      config: CONFIG,
+      providerLeaseId: "pc-abc",
+      leaseMetadata: leaseMetadata({ remoteCwd: "/workspace/custom" }),
+    });
+    expect(lease.metadata?.remoteCwd).toBe("/workspace/custom");
+  });
+
+   it("flags a resumed job-backend lease as native-sync-unsupported so the server keeps the base64 fallback", async () => {
+     h.clients = {
+       batch: {
@@ -0,0 +1,59 @@
+#!/bin/sh
+# Fetch upstream Paperclip at the pinned commit, apply our patches and make
+# the Dockerfile buildable with kaniko. Usage: prepare.sh <target-dir>
+set -eu
+HERE=$(cd "$(dirname "$0")" && pwd)
+. "$HERE/upstream.env"
+OUT=$1
+
+rm -rf "$OUT"
+git init -q "$OUT"
+cd "$OUT"
+git fetch -q --depth 1 https://github.com/paperclipai/paperclip.git "$PAPERCLIP_COMMIT"
+git -c advice.detachedHead=false checkout -q FETCH_HEAD
+test "$(git rev-parse HEAD)" = "$PAPERCLIP_COMMIT"
+
+for p in "$HERE"/*.patch; do
+  echo "applying $(basename "$p")"
+  git apply "$p"
+done
+
+# kaniko does not know `COPY --parents` (BuildKit only). Upstream uses it for
+# one line: every sandbox provider's package.json. Replace it with one plain
+# COPY per provider directory, which copies the same files to the same place.
+grep -c -- '^COPY --parents packages/plugins/sandbox-providers/\./\*/package.json packages/plugins/sandbox-providers/$' Dockerfile | grep -qx 1
+awk '
+  /^COPY --parents packages\/plugins\/sandbox-providers\// {
+    while (("ls -d packages/plugins/sandbox-providers/*/package.json" | getline f) > 0) {
+      d = f; sub(/package.json$/, "", d)
+      print "COPY " f " " d
+    }
+    next
+  }
+  { print }
+' Dockerfile > Dockerfile.kaniko
+if grep -q -- '--parents' Dockerfile.kaniko; then
+  echo "Dockerfile.kaniko still uses COPY --parents" >&2
+  exit 1
+fi
+
+# Build the Kubernetes sandbox provider into the image (REN-405). Upstream's
+# `production` target ships only its source; the plugin loader needs
+# dist/manifest.js. Same commands as upstream's `cloud-plugins` stage, run in
+# the `build` stage right after the server build, so production's
+# `COPY --from=build /app /app` carries dist/ and node_modules along. No extra
+# stage: kaniko would snapshot the whole build stage again for it.
+export ANCHOR='RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1)'
+grep -c -F -x -- "$ANCHOR" Dockerfile.kaniko | grep -qx 1
+K=packages/plugins/sandbox-providers/kubernetes
+awk -v k="$K" '
+  { print }
+  $0 == ENVIRON["ANCHOR"] {
+    print "RUN pnpm -C " k " install --ignore-workspace --no-lockfile \\"
+    print "  && pnpm -C " k " build \\"
+    print "  && test -f " k "/dist/manifest.js"
+  }
+' Dockerfile.kaniko > Dockerfile.kaniko.new
+mv Dockerfile.kaniko.new Dockerfile.kaniko
+grep -q -- "test -f $K/dist/manifest.js" Dockerfile.kaniko
+rm -rf .git
@@ -0,0 +1,7 @@
+# Upstream paperclipai/paperclip commit this image is built from (REN-401).
+# a027f76 is the master commit the nightly image ran when this was pinned, so
+# the only change against that image is the patch next to this file.
+# PAPERCLIP_VERSION is `git describe --tags --match 'v*' --long` of that
+# commit, the value upstream stamps into its own images. Bump both together.
+PAPERCLIP_COMMIT=a027f76a726e8a556674eded807c2de0e55f5cc0
+PAPERCLIP_VERSION=v2026.916.0-219-ga027f76a7
@@ -165,6 +165,21 @@ permanent recovery.

 ---

+## Velero restore-test: 80% OSD capacity gate
+
+The monthly Velero restore-test restores into the production Ceph pools. Before
+each schedule's restore it projects the restore's raw size (DataUpload
+`totalBytes` x the pool's replication/EC factor, spread over all OSDs) onto
+current OSD usage from Prometheus. If the fullest OSD would go above **80%**
+(`OSD_MAX_PCT` in `restore-test-cronjob.yaml`), that schedule is not restored:
+the Job log shows it as **SKIP** with the projected %, and the summary lists it
+under "Skipped (backup NOT restore-tested)". A SKIP does not fail the Job, so
+read the summary each month. If the projection cannot be made (Prometheus
+down, unsized DataUpload), the schedule is not restored and counts as FAIL.
+Background: REN-441, REN-442.
+
+---
+
 ## CNPG restore-test (verify backups are restorable — non-destructive)

 The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
@@ -70,7 +70,7 @@ spec:
              mountPath: /ca
      containers:
        - name: audiobookshelf
-          image: ghcr.io/advplyr/audiobookshelf:2.37.0 # {"$imagepolicy": "flux-system:audiobookshelf"}
+          image: ghcr.io/advplyr/audiobookshelf:2.37.1 # {"$imagepolicy": "flux-system:audiobookshelf"}
          ports:
            - name: http
              containerPort: 80
@@ -1,15 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: home-assistant-backup-password
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: home-assistant-backup-password # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/backup-password
@@ -1,7 +1,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - backup-password-externalsecret.yaml
  - pvcs.yaml
  - deployment.yaml
  - service.yaml
@@ -104,5 +103,13 @@ configMapGenerator:
    namespace: home-assistant
    files:
      - guard.sh=zigbee2mqtt-discovery-guard.sh
+  # Z2M external converters, copied into /app/data/external_converters by the
+  # seed-data init container on every start. Hash suffix disabled (name stays
+  # stable), so editing a converter does not auto-roll the pod — restart
+  # manually with `kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`.
+  - name: zigbee2mqtt-external-converters
+    namespace: home-assistant
+    files:
+      - wetterstation-ws90.js=zigbee2mqtt-ws90-converter.js
 generatorOptions:
  disableNameSuffixHash: true
@@ -132,6 +132,13 @@ spec:
              else
                echo "existing Z2M state present; NOT touching coordinator_backup.json"
              fi
+              # External converters: overwritten from git on EVERY start (not
+              # seed-only). Copied rather than mounted because Z2M writes a temp
+              # copy of each converter into this directory while loading it
+              # (dist/extension/externalJS.js), which a read-only mount refuses.
+              mkdir -p /app/data/external_converters
+              cp /external-converters/*.js /app/data/external_converters/
+              echo "copied external converters: $(ls /external-converters)"
          env:
            - name: MQTT_PASSWORD
              valueFrom:
@@ -147,6 +154,9 @@ spec:
            - name: coordinator-backup
              mountPath: /backup
              readOnly: true
+            - name: external-converters
+              mountPath: /external-converters
+              readOnly: true
          resources:
            requests:
              cpu: 10m
@@ -194,12 +204,15 @@ spec:
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 4
+          # Node sizes the V8 heap from this limit: 512Mi gave a 259 MB heap,
+          # and Z2M died on "JavaScript heap out of memory" (exit 134) at
+          # 2026-09-28 17:56Z and 2026-09-29 07:36Z.
          resources:
            requests:
              cpu: 50m
-              memory: 128Mi
+              memory: 256Mi
            limits:
-              memory: 512Mi
+              memory: 1Gi
        # Fires a forced interview whenever a still-un-interviewed device
        # transmits — i.e. at the only moment a sleepy battery device can
        # actually complete one. Z2M itself only retries interviews on a ZDO
@@ -337,6 +350,9 @@ spec:
        - name: config-template
          configMap:
            name: zigbee2mqtt-config
+        - name: external-converters
+          configMap:
+            name: zigbee2mqtt-external-converters
        - name: coordinator-backup
          secret:
            secretName: zigbee2mqtt-coordinator-backup
@@ -0,0 +1,48 @@
+// Wetterstation (WS90) = the built-in zigbee-herdsman-converters definition
+// plus one extra configure step that applies the reporting configuration from
+// zigbee2mqtt-ws90-reporting.md. Z2M runs configure after every interview and
+// on bridge/request/device/configure, so those no longer revert the device to
+// the converter defaults (as happened 2026-09-24..28).
+//
+// The extra step runs after the built-in ones (extend order), so its values
+// win. `version` makes Z2M reconfigure once when this file is first loaded:
+// it re-runs configure when definition.version differs from the device's
+// meta.configured. Bump it whenever REPORTING changes.
+const {definitions} = require("zigbee-herdsman-converters/devices/shelly");
+
+const base = definitions.find((d) => d.model === "WS90");
+if (!base) throw new Error("WS90 definition not found in zigbee-herdsman-converters/devices/shelly");
+
+const SHELLY = 5264;
+
+// [cluster, attribute, reportable_change, manufacturerCode]
+const REPORTING = [
+    ["msTemperatureMeasurement", "measuredValue", 10],
+    ["msRelativeHumidity", "measuredValue", 50],
+    ["msPressureMeasurement", "measuredValue", 1],
+    ["msIlluminanceMeasurement", "measuredValue", 5],
+    ["shellyWS90Wind", "windSpeed", 1, SHELLY],
+    ["shellyWS90Wind", "windDirection", 1, SHELLY],
+    ["shellyWS90Wind", "gustSpeed", 1, SHELLY],
+    ["shellyWS90UV", "uvIndex", 1, SHELLY],
+    ["shellyWS90Rain", "rainStatus", 1, SHELLY],
+    ["shellyWS90Rain", "precipitation", 1, SHELLY],
+];
+
+const reporting = {
+    isModernExtend: true,
+    configure: [
+        async (device) => {
+            const endpoint = device.getEndpoint(1);
+            for (const [cluster, attribute, change, manufacturerCode] of REPORTING) {
+                await endpoint.configureReporting(
+                    cluster,
+                    [{attribute, minimumReportInterval: 5, maximumReportInterval: 3600, reportableChange: change}],
+                    manufacturerCode ? {manufacturerCode} : {},
+                );
+            }
+        },
+    ],
+};
+
+module.exports = {...base, version: "ws90-reporting-1", extend: [...base.extend, reporting]};
@@ -2,10 +2,15 @@

 Records the ZCL attribute-reporting configuration applied to the Wetterstation
 (Ecowitt WS90 powered by Shelly, IEEE `0xc02cedfffe958f40`) and how to restore
-it, because **this configuration lives on the device and in Z2M's
-`database.db`, not in git**. Zigbee2MQTT has no settings-schema field for
-reporting configuration, so there is nothing declarative to reconcile — Flux
-cannot restore it and a fresh PVC will not have it.
+it. The configuration itself lives on the device and in Z2M's `database.db`.
+
+**Since 2026-10-01 it is applied from git:** the external converter
+`zigbee2mqtt-ws90-converter.js` extends the built-in WS90 definition with a
+configure step that sets the values below, so every interview and every
+`bridge/request/device/configure` applies them instead of the converter
+defaults. The manual commands under "Re-applying" are only needed if the
+converter is not loaded. Keep the table and `REPORTING` in the converter in
+sync, and bump the converter's `version` when they change.

 ## Applied configuration

@@ -71,14 +76,16 @@ the modernExtend value is the outlier.

 ## When this is lost

-Anything that makes Z2M re-run `configure` on the device overwrites all of it
-with the converter defaults:
+Without the external converter, anything that makes Z2M re-run `configure` on
+the device overwrites all of it with the converter defaults:

- re-pairing or re-joining the device
+- re-pairing, re-joining or re-interviewing the device
 - an explicit `zigbee2mqtt/bridge/request/device/configure`
 - a converter update that changes the definition and triggers reconfiguration

-It also will not exist on a fresh PVC. Re-apply with the commands below.
+With the converter loaded, those same events apply the values above. Check
+that it is loaded: Z2M logs `Successfully configured 'Wetterstation'
+(definition vws90-reporting-1)` after a configure.

 ## Re-applying

@@ -29,6 +29,17 @@
 # been dropped and the metric would have stayed absent. The matching ingress rule
 # is added to infrastructure/network-policies/external-secrets-ns.yaml in the
 # same pull request. Without it this file is decoration.
+#
+# NO TLS AND NO AUTH ON THIS SCRAPE, and that is the chart's posture rather than
+# an omission here. In 2.11.0 `metrics.listen.secure.enabled` and
+# `metrics.listen.auth.enabled` are both false (values.yaml:365-369) and
+# infrastructure/external-secrets/helmrelease.yaml overrides neither, so the
+# controller serves /metrics over plain HTTP to anything that can reach :8080.
+# Hence no `scheme`, no `tlsConfig` and no `bearerTokenSecret` below: adding
+# them would break the scrape, not harden it. What limits exposure is the
+# network policy — a single port from a single namespace — so turning on
+# `secure.enabled` here would mean a Helm upgrade of ESO to close a hole the
+# CiliumNetworkPolicy already closes.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
@@ -11,6 +11,31 @@ spec:
  groups:
    - name: disk-wear.rules
      rules:
+        # The drives carrying Talos's EPHEMERAL volume (/var), as (node, device)
+        # in smartctl-exporter's labels. EPHEMERAL sits on retired Ceph drives
+        # whose contents rebuild, and the owner's call is that they run to
+        # failure rather than be replaced on wear (REN-419, REN-420). So the two
+        # wear alerts below skip them. The failure alerts — spare, critical
+        # warning, media errors — still cover them: run to failure still means
+        # knowing when it failed.
+        #
+        # Resolved from node-exporter's /var mount, not a list of serials, so it
+        # follows a drive swap without an edit. If the join stops matching (a
+        # relabel, an encrypted /var on a dm device) the series is empty, and
+        # the wear alerts return for EPHEMERAL drives: it fails loud, not
+        # silent.
+        - record: node_device:nvme_ephemeral_volume:info
+          expr: >-
+            count by (node, device) (
+              label_replace(
+                label_replace(
+                  node_filesystem_size_bytes{mountpoint="/var"}
+                    * on (instance) group_left (nodename) node_uname_info,
+                  "node", "$1", "nodename", "(.+)"
+                ),
+                "device", "$1", "device", "/dev/(nvme[0-9]+)n[0-9]+(p[0-9]+)?"
+              )
+            )
        # Wear percentages are self-reported: 100 means the rated endurance is
        # spent, not that the drive stops.
        #
@@ -45,6 +70,7 @@ spec:
              )
            ) < 180
            and smartctl_device_percentage_used >= 80
+            unless on (node, device) node_device:nvme_ephemeral_volume:info
          # Long, because a projection off a 7d rate should not react to a day.
          for: 6h
          labels:
@@ -55,13 +81,17 @@ spec:
              The drive is past 80% of its rated endurance and the write rate
              of the last week would spend the rest within six months. This is
              a lead-time warning, not a fault: decide whether to replace the
-              drive or move what is writing to it. The Disk Health dashboard
+              drive or move what is writing to it. Drives carrying EPHEMERAL
+              are excluded; they run to failure. The Disk Health dashboard
              shows which drives carry which role.
        # Kept as a flat threshold on purpose: past rated endurance the drive
        # runs outside what the manufacturer stands behind regardless of how
-        # fast it got there, and the alert clears by replacing it.
+        # fast it got there, and the alert clears by replacing it. EPHEMERAL
+        # drives are excluded for the same reason as above.
        - alert: NvmeDriveWearCritical
-          expr: smartctl_device_percentage_used >= 95
+          expr: >-
+            smartctl_device_percentage_used >= 95
+            unless on (node, device) node_device:nvme_ephemeral_volume:info
          for: 1h
          labels:
            severity: critical
@@ -30,6 +30,7 @@ retired the depth rule in favour of one persistence rule (`> 5` for `20m`).
 | `external-secrets_test.yaml` | Hand-written suite for `prometheusrule-external-secrets.yaml` (REN-103). |
 | `home-assistant_test.yaml` | Hand-written suite for `prometheusrule-home-assistant.yaml` (REN-245). |
 | `ceph_test.yaml` | Hand-written suite for the `CephPoolLostRedundancyBlind` canary in `prometheusrule-ceph.yaml` (REN-93). |
+| `disk-wear_test.yaml` | Hand-written suite for the EPHEMERAL exclusion in `prometheusrule-disk-wear.yaml` (REN-419). |
 | `mutations.txt` | Negative controls. Each must make the suite fail. |

 ## Adding a suite
@@ -0,0 +1,124 @@
+# promtool test rules for the EPHEMERAL exclusion in prometheusrule-disk-wear.yaml
+# (REN-419).
+#
+# Run by the `promtool` step of .woodpecker/ci-manifests-test.yml through
+# scripts/ci/promtool-rules.sh, which strips `spec:` from the manifest into a
+# mirrored tree — hence `../` below, resolved from this file's directory. See
+# README.md here. The negative controls for this suite are the `disk-wear-*`
+# entries in mutations.txt.
+#
+# WHAT IT IS FOR. The two wear alerts skip the drive that carries /var, found
+# by joining node-exporter's /var mount onto smartctl-exporter's (node, device).
+# Each case puts an EPHEMERAL drive and a non-EPHEMERAL drive side by side with
+# identical SMART data, so the only thing that can separate them is the join.
+#
+# Fixture shape: smartctl-exporter labels drives `nvme0`, `nvme1` and carries
+# `node` from the ServiceMonitor relabel; node-exporter carries `instance` and
+# reports /var on the partition, `/dev/nvme1n1p1`.
+rule_files:
+  - ../prometheusrule-disk-wear.yaml
+evaluation_interval: 1m
+tests:
+  # A. Projection. k8s-1 nvme0 and nvme1 wear identically: 89%, 1.5e9 bytes
+  #    per 5m (4.32e11 a day), 9e14 lifetime bytes at 192h. That is 0.04272 % a
+  #    day, 6 % to go, 140 days — under 180. Only nvme0 may fire; nvme1
+  #    carries /var. Neither is at 95%, so the critical alert stays quiet.
+  #
+  #    Sampled at 5m, not 1h: the instant selectors look back 5m, so hourly
+  #    samples leave the series absent for most of each hour and the 6h `for`
+  #    never completes. Not 1m either: a 7d rate over 1m samples exceeds
+  #    promtool's query sample limit.
+  - interval: 5m
+    input_series:
+      - series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme0",instance="10.0.0.1:9633"}'
+        values: '89+0x2320'
+      - series: 'smartctl_device_bytes_written{node="k8s-1",device="nvme0",instance="10.0.0.1:9633"}'
+        values: '896544000000000+1500000000x2320'
+      - series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
+        values: '89+0x2320'
+      - series: 'smartctl_device_bytes_written{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
+        values: '896544000000000+1500000000x2320'
+      - series: 'node_filesystem_size_bytes{instance="10.0.0.1:9100",mountpoint="/var",device="/dev/nvme1n1p1",fstype="xfs"}'
+        values: '1000000000000+0x2320'
+      - series: 'node_uname_info{instance="10.0.0.1:9100",nodename="k8s-1"}'
+        values: '1+0x2320'
+    alert_rule_test:
+      - eval_time: 192h
+        alertname: NvmeDriveWearProjectedExhaustion
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              node: k8s-1
+              device: nvme0
+              instance: 10.0.0.1:9633
+            exp_annotations:
+              summary: "NVMe nvme0 on 10.0.0.1:9633 projects to 95% wear in 140 days"
+              description: "The drive is past 80% of its rated endurance and the write rate of the last week would spend the rest within six months. This is a lead-time warning, not a fault: decide whether to replace the drive or move what is writing to it. Drives carrying EPHEMERAL are excluded; they run to failure. The Disk Health dashboard shows which drives carry which role."
+      - eval_time: 192h
+        alertname: NvmeDriveWearCritical
+        exp_alerts: []
+
+  # B. Critical, two nodes. Every drive is at 96%. On k8s-1 /var is nvme1; on
+  #    k8s-2 it is nvme2, which reports no SMART here. So k8s-1 nvme1 is the only
+  #    one excluded, and k8s-2 nvme1 must still fire — the exclusion is per
+  #    node, not per device name.
+  - interval: 1m
+    input_series:
+      - series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme0",instance="10.0.0.1:9633"}'
+        values: '96+0x200'
+      - series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
+        values: '96+0x200'
+      - series: 'smartctl_device_percentage_used{node="k8s-2",device="nvme1",instance="10.0.0.2:9633"}'
+        values: '96+0x200'
+      - series: 'node_filesystem_size_bytes{instance="10.0.0.1:9100",mountpoint="/var",device="/dev/nvme1n1p1",fstype="xfs"}'
+        values: '1000000000000+0x200'
+      - series: 'node_uname_info{instance="10.0.0.1:9100",nodename="k8s-1"}'
+        values: '1+0x200'
+      - series: 'node_filesystem_size_bytes{instance="10.0.0.2:9100",mountpoint="/var",device="/dev/nvme2n1p1",fstype="xfs"}'
+        values: '1000000000000+0x200'
+      - series: 'node_uname_info{instance="10.0.0.2:9100",nodename="k8s-2"}'
+        values: '1+0x200'
+    alert_rule_test:
+      - eval_time: 120m
+        alertname: NvmeDriveWearCritical
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              node: k8s-1
+              device: nvme0
+              instance: 10.0.0.1:9633
+            exp_annotations:
+              summary: "NVMe nvme0 on 10.0.0.1:9633 is at 96% rated wear"
+              description: "The drive is at or past the end of its rated endurance. It will likely keep working for a while, but it is now operating outside what the manufacturer stands behind, and replacement should be scheduled rather than considered."
+          - exp_labels:
+              severity: critical
+              node: k8s-2
+              device: nvme1
+              instance: 10.0.0.2:9633
+            exp_annotations:
+              summary: "NVMe nvme1 on 10.0.0.2:9633 is at 96% rated wear"
+              description: "The drive is at or past the end of its rated endurance. It will likely keep working for a while, but it is now operating outside what the manufacturer stands behind, and replacement should be scheduled rather than considered."
+
+  # C. Fails loud. /var is on a device the join cannot map to an NVMe name (an
+  #    encrypted volume on dm-0). The exclusion then matches nothing and the
+  #    EPHEMERAL drive alerts again, which is the intended direction to fail.
+  - interval: 1m
+    input_series:
+      - series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
+        values: '96+0x200'
+      - series: 'node_filesystem_size_bytes{instance="10.0.0.1:9100",mountpoint="/var",device="/dev/dm-0",fstype="xfs"}'
+        values: '1000000000000+0x200'
+      - series: 'node_uname_info{instance="10.0.0.1:9100",nodename="k8s-1"}'
+        values: '1+0x200'
+    alert_rule_test:
+      - eval_time: 120m
+        alertname: NvmeDriveWearCritical
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              node: k8s-1
+              device: nvme1
+              instance: 10.0.0.1:9633
+            exp_annotations:
+              summary: "NVMe nvme1 on 10.0.0.1:9633 is at 96% rated wear"
+              description: "The drive is at or past the end of its rated endurance. It will likely keep working for a while, but it is now operating outside what the manufacturer stands behind, and replacement should be scheduled rather than considered."
@@ -149,3 +149,21 @@ retention-no-size-clause|kubernetes/apps/monitoring-rules/prometheusrule-prometh
 retention-ratio-0.9|kubernetes/apps/monitoring-rules/prometheusrule-prometheus-retention.yaml|s/^\([[:space:]]*\)> 0.8$/\1> 0.9/
 retention-lookback-1d|kubernetes/apps/monitoring-rules/prometheusrule-prometheus-retention.yaml|s/)\[7d:1h\]$/)[1d:1h]/
 retention-cap-at-peak-time|kubernetes/apps/monitoring-rules/prometheusrule-prometheus-retention.yaml|s/\/ max(prometheus_tsdb_retention_limit_bytes{job="kube-prometheus-stack-prometheus"})$/\/ min_over_time(max(prometheus_tsdb_retention_limit_bytes{job="kube-prometheus-stack-prometheus"})[7d:1h])/
+
+# prometheusrule-disk-wear.yaml (REN-419): the EPHEMERAL exclusion on the two
+# wear alerts. Detected by the cases in disk-wear_test.yaml.
+#
+# Gate inverted on the projection: the EPHEMERAL drive alone fires.
+# Detected by: case A.
+disk-wear-projection-gate-inverted|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|/alert: NvmeDriveWearProjectedExhaustion/,/for: 6h/s/unless on (node, device)/and on (node, device)/
+# Gate inverted on the critical rule.
+# Detected by: case B.
+disk-wear-critical-gate-inverted|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|/alert: NvmeDriveWearCritical/,/for: 1h/s/unless on (node, device)/and on (node, device)/
+# Matched on device name only: k8s-1's /var on nvme1 then silences nvme1 on
+# every node.
+# Detected by: case B, k8s-2 nvme1.
+disk-wear-critical-device-only|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|/alert: NvmeDriveWearCritical/,/for: 1h/s/unless on (node, device)/unless on (device)/
+# Partition suffix not stripped: /dev/nvme1n1p1 no longer maps to nvme1 and the
+# exclusion matches nothing.
+# Detected by: cases A and B.
+disk-wear-ephemeral-keeps-partition|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|s/n\[0-9\]+(p\[0-9\]+)?"/n[0-9]+"/
@@ -6,13 +6,12 @@
 # 5-album spot check), so beets now writes tags. Art plugins stay off: they
 # would add or embed cover files, a separate change.
 #
-# Retry trial: the write run matched 74 of 560 albums. 486 were skipped: most
-# folders hold only part of an album, and beets caps a match with missing
-# tracks below "strong", so quiet mode skipped them, and incremental import
-# never looked at them again. This trial retries them with a partial-album
-# match allowed, but imports with -W (no tag write) and skips `beet write`,
-# so no file changes until the new matches are checked. The 74 written albums
-# are re-matched too; the run lists any whose release changed.
+# Retry: the first write run matched 74 of 560 albums; most skipped folders
+# hold only part of an album. Partial-album matches are allowed now (see
+# max_rec). A -W trial matched 158 albums, all at 96% or better, and none of
+# the 74 written albums changed release, so beets now writes them. Muse "Black
+# Holes and Revelations" matched an instrumental release: it is ignored and
+# dropped from the DB (DB only, no file change).
 #
 # Lidarr does not write tags: lidarr-reconcile sets writeAudioTags "no" on
 # every pod start (lidarr.yaml), so it cannot undo the beets tags.
@@ -42,6 +41,9 @@ data:
    library: /config/library.db
    statefile: /config/state.pickle
    plugins: [musicbrainz, fetchart, embedart, lastgenre]
+    # Beets defaults plus folders whose only match is a wrong release.
+    ignore: ['.*', '*~', 'System Volume Information', 'lost+found',
+             'Black Holes and Revelations']
    # Keep the first-release date in year; reissues must not change it.
    original_date: yes
    match:
@@ -95,27 +97,39 @@ data:
      rm -f /config/state.pickle
      touch /config/.retry-ren350
    fi
+    # One-time: the trial stored an instrumental release for this album.
+    # `rm` without -d drops the DB entry only; the file is not touched.
+    if [ ! -f /config/.drop-ren350 ]; then
+      beet rm -a -f 'path:/music/Muse/Black Holes and Revelations'
+      touch /config/.drop-ren350
+    fi
    beet ls -a -f '$path $mb_albumid' | sort > /config/logs/albums-before.txt
    LOG=/config/logs/import-$(date +%Y%m%d-%H%M).log
-    # Trial: -W stores matches in the DB but writes no file.
-    beet import -W -l "$LOG" /music
+    beet import -l "$LOG" /music
    rc=$?
    beet ls -a -f '$path $mb_albumid' | sort > /config/logs/albums-after.txt
    comm -23 /config/logs/albums-before.txt /config/logs/albums-after.txt \
      > /config/logs/albums-changed.txt
    echo "== import exit $rc"
-    echo "== written albums with a changed release: $(wc -l < /config/logs/albums-changed.txt)"
+    echo "== albums with a changed release: $(wc -l < /config/logs/albums-changed.txt)"
    cat /config/logs/albums-changed.txt
    echo "== albums in library: $(beet ls -a | wc -l)"
    echo "== skipped this run: $(grep -c '^skip ' "$LOG")"
    echo "== duplicates skipped: $(grep -c '^duplicate-skip ' "$LOG")"
-    # Trial: preview only. `beet write` comes back once the matches are checked.
+    # The trial (-W) stored matches in the DB only; `beet write` puts them
+    # into the files (tags only, no move).
    beet write -p > /config/logs/write-preview.txt 2>&1
    echo "== items to write: $(grep -vc '^ ' /config/logs/write-preview.txt)"
+    beet write
+    wrc=$?
+    echo "== write exit $wrc"
+    beet write -p > /config/logs/write-after.txt 2>&1
+    echo "== items still differing after write: $(grep -vc '^ ' /config/logs/write-after.txt)"
    echo "== skipped albums"
    grep '^skip ' "$LOG"
-    echo "== changes the write would make (first 2000 lines)"
+    echo "== written changes (first 2000 lines)"
    head -n 2000 /config/logs/write-preview.txt
+    [ "$rc" -eq 0 ] && rc=$wrc
    exit $rc
 ---
 apiVersion: batch/v1
@@ -171,10 +185,9 @@ spec:
                - name: beets-config
                  mountPath: /etc/beets
                  readOnly: true
-                # Read-only for the retry trial; writable again with `beet write`.
+                # Writable for tag writes; move/copy/delete stay off above.
                - name: music
                  mountPath: /music
-                  readOnly: true
              resources:
                requests:
                  cpu: 100m
@@ -0,0 +1,214 @@
+---
+# Per-agent kubectl identities for agent pods (REN-412, parent REN-284).
+#
+# Each agent gets its own ServiceAccount instead of the shared pod token of
+# `paperclip-prometheus-read`. Atlas, Iris, Hestia and Argus get exactly what
+# they hold today through that shared account: built-in `view` cluster-wide
+# (no Secrets, no write verb) plus the `monitoring/paperclip-prometheus-read`
+# Role (see prometheus-read-rbac.yaml). Hermes gets only `get` on the five
+# token Secrets below, to issue and rotate the kubeconfigs.
+#
+# Token Secrets carry no data in git; kube-controller-manager fills
+# `data.token`. Flux applies server-side and does not own `data`, so it does
+# not wipe the token. Revoke one agent: delete its Secret (the token dies at
+# once; Flux recreates it with a fresh token on the next reconcile).
+#
+# The shared account may read the tokens too (Hermes still runs in the shared
+# pod). Each token is worth at most what the shared account already holds, so
+# that is no escalation. Drop that subject once Hermes runs in its own pod.
+#
+# PRODUCTION
+#
+# ServiceAccounts, bindings and empty token Secrets only. No workload changes,
+# nothing restarts. Revert: delete this file and its line in kustomization.yaml.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-atlas
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-iris
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-hestia
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-argus
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-hermes
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-atlas-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-atlas
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-iris-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-iris
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-hestia-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-hestia
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-argus-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-argus
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-hermes-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-hermes
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: paperclip-agents-view
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: agent-atlas
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-iris
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-hestia
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-argus
+    namespace: paperclip
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: paperclip-agents-prometheus-read
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  # Existing Role in prometheus-read-rbac.yaml, unchanged.
+  name: paperclip-prometheus-read
+# Subjects live in `paperclip`, the binding in `monitoring`: `namespace:` is
+# load-bearing.
+subjects:
+  - kind: ServiceAccount
+    name: agent-atlas
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-iris
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-hestia
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-argus
+    namespace: paperclip
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: agent-token-read
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get"]
+    resourceNames:
+      - agent-atlas-token
+      - agent-iris-token
+      - agent-hestia-token
+      - agent-argus-token
+      - agent-hermes-token
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: agent-token-read
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: agent-token-read
+subjects:
+  # Hermes still runs in the shared pod; drop this subject once Hermes runs in
+  # its own pod.
+  - kind: ServiceAccount
+    name: paperclip-prometheus-read
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-hermes
+    namespace: paperclip
@@ -10,6 +10,7 @@ resources:
  - prometheus-read-rbac.yaml
  - cluster-view-rbac.yaml
  - sandbox-rbac.yaml
+  - agent-kubectl-rbac.yaml
  - paperclip.yaml
  - gate-service.yaml
  - ingress.yaml
@@ -34,7 +34,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: paperclip
-          image: ghcr.io/paperclipai/paperclip:nightly@sha256:c02c7a8385fe5e3e9999ec03ecb45f29d60b8bf44dac3fedf032d7ef2046045d # {"$imagepolicy": "flux-system:paperclip"}
+          image: gitea.renner.uno/konairius/paperclip:build.1790806287@sha256:f0b4b4e09cc02fbc472faedf3edb6d39339ade7e2932c6a435f601209b92d646 # {"$imagepolicy": "flux-system:paperclip"}
          ports:
            - name: http
              containerPort: 3100
@@ -134,7 +134,7 @@ spec:
        # is a device-bricking class of risk. Bumps are a reviewed, manual
        # act — see kubernetes/infrastructure/image-automation/README.md.
        - name: terminus
-          image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.76.0 # {"$imagepolicy": "flux-system:terminus"}
          ports:
            - name: http
              containerPort: 2345
@@ -197,7 +197,7 @@ spec:
            capabilities:
              drop: ["ALL"]
        - name: sidekiq
-          image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.76.0 # {"$imagepolicy": "flux-system:terminus"}
          command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
          envFrom:
            - configMapRef:
@@ -937,7 +937,7 @@ subjects:
 # with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
 stringData:
-  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
+  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.1 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
 kind: Secret
 metadata:
  name: plugin-barman-cloud-kg5c2795f7
@@ -998,7 +998,7 @@ spec:
            secretKeyRef:
              key: SIDECAR_IMAGE
              name: plugin-barman-cloud-kg5c2795f7
-        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
+        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.1 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
        name: barman-cloud
        ports:
        - containerPort: 9090
@@ -1,13 +1,18 @@
+# Our own build (REN-401): upstream master at a pinned commit plus the k8s
+# sandbox fix, built by .woodpecker/build-paperclip.yml. Back to
+# ghcr.io/paperclipai/paperclip:nightly once upstream PR #13733 is merged;
+# see ci/images/paperclip/README.md.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 metadata:
  name: paperclip
  namespace: flux-system
 spec:
-  image: ghcr.io/paperclipai/paperclip
-  interval: 1h
+  image: gitea.renner.uno/konairius/paperclip
+  interval: 10m
 ---
-# Upstream publishes no semver tags; track the nightly tag by digest.
+# The pipeline tags every build `build.<unix-ts>`; latest build wins. Pinned by
+# digest in the manifest like before.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImagePolicy
 metadata:
@@ -17,9 +22,10 @@ spec:
  imageRepositoryRef:
    name: paperclip
  digestReflectionPolicy: Always
-  interval: 1h
+  interval: 10m
  filterTags:
-    pattern: '^nightly$'
+    pattern: '^build\.(?P<ts>\d+)$'
+    extract: '$ts'
  policy:
-    alphabetical:
+    numerical:
      order: asc
@@ -16,13 +16,29 @@ spec:
    # dropped and `externalsecret_status_condition` held 0 series.
    #
    # Port-restricted, unlike the equivalent rule in velero-ns.yaml, which admits
-    # `monitoring` to the whole namespace. 8080 is the metrics port and the only
-    # thing `monitoring` has any business reaching here; the webhook (10250) and
-    # the health port stay closed to it. 8080 is the chart default
+    # `monitoring` to the whole namespace. Namespace-wide is the norm and this is
+    # the sole exception: counted per ingress list item, this directory holds 20
+    # `monitoring` grants across 19 files (influxdb-ns.yaml carries two — one
+    # whole-namespace, one Grafana-pod), and this rule is the only one of the 20
+    # with a `toPorts`. home-assistant-ns.yaml reads as port-scoped and is not —
+    # the `toPorts` below its `monitoring` rule belongs to the `evcc` rule that
+    # follows.
+    # 8080 is the metrics port and the only thing `monitoring` has any business
+    # reaching here; the webhook (10250) and the health ports (8082 on the
+    # controller, 8081 on the other two) stay closed to it. 8080 is the chart
+    # default
    # (`metrics.listen.port`) and helmrelease.yaml does not override it — if that
    # is ever overridden, this port and the PodMonitor's named `metrics` endpoint
    # both have to follow.
    #
+    # Scoped by PORT, not by pod: this CNP's `endpointSelector` is {}, so the
+    # grant reaches all three of the chart's deployments — controller, webhook,
+    # cert-controller — which all serve metrics on 8080. Only the controller is
+    # scraped today (the PodMonitor's selector is exact on the unsuffixed name),
+    # so the extra two are reachable-but-unvisited. Deliberate: adding a monitor
+    # for either later needs no policy change, and neither exposes more than
+    # controller-runtime internals on that port.
+    #
    # The consumer is apps/monitoring-rules/external-secrets-podmonitor.yaml.
    # Additive: this rule cannot deny anything previously allowed, so ESO's
    # existing reachability to OpenBao and the apiserver is unchanged.
@@ -7,6 +7,7 @@ resources:
  - authentik-ns.yaml
  - paperless-ns.yaml
  - paperclip-ns.yaml
+  - paperclip-renner-ns.yaml
  - audiobookshelf-ns.yaml
  - home-assistant-ns.yaml
  - spegel-ns.yaml
@@ -50,6 +50,17 @@ spec:
        - ports:
            - port: "9090"
              protocol: TCP
+    # velero restore-test Job: its capacity gate reads OSD usage from
+    # Prometheus :9090 (restore-test.sh, preflight_capacity). Scoped to the
+    # Job's pod label; the velero namespace has no egress policy.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": velero
+            app.kubernetes.io/name: restore-test
+      toPorts:
+        - ports:
+            - port: "9090"
+              protocol: TCP
    # Kubelet probes + control-plane (PrometheusRule/ServiceMonitor admission
    # webhooks via kube-prometheus-stack-operator are called by kube-apiserver).
    # Also covers external alloy-syslog traffic — ETP=Cluster on the LB SNATs
@@ -0,0 +1,70 @@
+# Agent pods in paperclip-renner (REN-284) reach Gitea through Traefik (REN-411).
+#
+# gitea.renner.uno resolves to the Traefik LB VIP (fd99::86 / 10.2.99.86).
+# Cilium translates the VIP to a Traefik pod before egress policy, so the
+# destination is a cluster endpoint, not `world`, and the plugin's `toFQDNs`
+# rule for gitea.renner.uno never matches it. Allow the Traefik pods by
+# label instead, on the post-DNAT port: service 443 -> targetPort websecure
+# = container port 8443.
+#
+# Not labelled `paperclip.io/managed-by`, so the Paperclip k8s plugin does not
+# own or delete it. Cilium unions allows, so this only adds to
+# paperclip-egress-fqdn and paperclip-deny-all. Traefik's side is open:
+# ingress-default-deny (ingress-isolation.yaml) admits `cluster`.
+#
+# Revert: delete this file and its line in kustomization.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: agent-egress-traefik
+  namespace: paperclip-renner
+spec:
+  endpointSelector:
+    matchLabels:
+      paperclip.io/role: agent
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP
+---
+# Agent pods reach the Kubernetes API server with their own kubeconfig
+# (REN-412, parent REN-284; RBAC in apps/paperclip/agent-kubectl-rbac.yaml).
+#
+# A `toCIDRSet` rule cannot open this path: `policy-cidr-match-mode` is unset
+# in cilium-config, so CIDR rules never match node IPs, and the API server
+# (10.2.46.11-13:6443, Service 10.96.0.1:443 translated by KPR) runs on the
+# nodes. The `kube-apiserver` entity matches it. Port is the post-DNAT 6443.
+#
+# Selects by agent id, so only the five agents with a kubectl identity get
+# the path; other agent pods stay closed. Not labelled
+# `paperclip.io/managed-by`, so the plugin does not own or delete it.
+#
+# Revert: delete this document.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-agents-kube-apiserver
+  namespace: paperclip-renner
+spec:
+  endpointSelector:
+    matchExpressions:
+      - key: paperclip.io/agent-id
+        operator: In
+        values:
+          - 27d813a4-ae75-4913-b25b-3516a893d614 # Atlas
+          - 6eb4d648-83b4-40cb-b16e-d05d3ef372cb # Iris
+          - 32b3c62a-a5dc-4499-b970-25f5c7f60e8c # Hestia
+          - b023e112-1466-46f1-9fc1-e87e5cd3f005 # Argus
+          - b1be203c-d260-4922-8b48-ab9e87e73fb0 # Hermes
+  egress:
+    - toEntities:
+        - kube-apiserver
+      toPorts:
+        - ports:
+            - port: "6443"
+              protocol: TCP
@@ -196,9 +196,10 @@ spec:
  # several apps under apps/ set `namespace: home-assistant` in their
  # Kustomization, so their PVCs land here too.
  #
-  # home-assistant-config has its own restore path: the HA pod's `seed`
-  # initContainer pulls the native HAOS backup. Layering a Velero PVC restore
-  # on top of that for the same PVC needs its own review.
+  # This is the ONLY restore path for home-assistant-config (.storage,
+  # secrets.yaml, the config git checkout). The HAOS-backup `seed`
+  # initContainer that used to repopulate it was removed in 21a0a43f.
+  # Restore procedure and test: REN-249 runbook section C.2.
  schedule: "45 6 * * *"
  useOwnerReferencesInBackup: false
  template:
@@ -52,6 +52,11 @@ spec:
      activeDeadlineSeconds: 21600  # 6 h hard cap (immich alone can take ~3 h)
      ttlSecondsAfterFinished: 3456000  # 40 d — enough to inspect after the next monthly run
      template:
+        metadata:
+          labels:
+            # Matched by monitoring-ns.yaml: the capacity gate reads OSD
+            # usage from Prometheus :9090.
+            app.kubernetes.io/name: restore-test
        spec:
          serviceAccountName: restore-test
          restartPolicy: Never
@@ -73,6 +78,15 @@ spec:
              env:
                - name: RESTORE_PHASE_TIMEOUT_S
                  value: "14400"
+                # Whole-sweep budget; caps each restore's timeout. Keep
+                # ~30 min below activeDeadlineSeconds so the summary and
+                # cleanup run before Kubernetes kills the pod.
+                - name: SWEEP_BUDGET_S
+                  value: "19800"
+                # Skip a restore projected to take the fullest OSD past
+                # this (SKIP, not FAIL). See restore-test.sh.
+                - name: OSD_MAX_PCT
+                  value: "80"
                # readOnlyRootFilesystem=true blocks kubectl from writing
                # discovery cache to ~/.kube. Point HOME at the writable
                # tmp emptyDir so kubectl works.
@@ -25,15 +25,31 @@
 # multiple hours (millions of small files), so the full sweep is sized
 # at up to 6 h via the cronjob's activeDeadlineSeconds.
 #
+# Capacity gate: before each restore, the raw size of the restore is
+# projected onto the fullest OSD (see preflight_capacity). A schedule whose
+# projected peak is above OSD_MAX_PCT (80%) is not restored and is reported
+# as SKIP, not FAIL. Skips are listed in the summary so an untested backup
+# stays visible.
+#
 # Exit non-zero if any check fails. Prints a summary at the end.

 set -uo pipefail

 PASS=0
 FAIL=0
+SKIP=0
 FAILURES=()
+SKIPS=()
 RUN_ID=$(date -u +%s)
 RESTORE_PHASE_TIMEOUT_S=${RESTORE_PHASE_TIMEOUT_S:-14400}   # 4 h — immich library (~180 GiB, millions of small files) does not finish in 2 h
+# Whole-sweep budget, kept below the Job's activeDeadlineSeconds. The
+# 2026-10-01 run started immich 2 h 17 m in, so its 4 h phase timeout could
+# never fire before the 6 h deadline killed the pod: no summary, no
+# cleanup. Each restore's timeout is capped by what is left of this budget,
+# and services that start after it is used up are SKIPped. The slack to the
+# deadline covers the last service's parity probes, cleanup and summary.
+SWEEP_BUDGET_S=${SWEEP_BUDGET_S:-19800}   # 5 h 30 m
+SWEEP_DEADLINE=$(( $(date +%s) + SWEEP_BUDGET_S ))

 log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }

@@ -45,11 +61,74 @@ check() {
        PASS=$((PASS + 1))
    else
        echo "FAIL"
+        # Also printed now: a pod killed before the summary loses FAILURES.
+        tail -4 /tmp/check.out | sed 's/^/      | /'
        FAILURES+=("$name -- $(tail -4 /tmp/check.out | tr '\n' ' ')")
        FAIL=$((FAIL + 1))
    fi
 }

+skip() {
+    local name="$1" reason="$2"
+    printf '  %-72s SKIP\n' "$name"
+    SKIPS+=("$name -- $reason")
+    SKIP=$((SKIP + 1))
+}
+
+# Capacity gate (REN-441, REN-442). Every restore lands on the production
+# Ceph pools, so a large one pushes the OSDs toward nearfull while it is
+# alive. On 2026-10-01 audiobookshelf-books (87 GB, EC 2+1) took the
+# fullest OSD from 74.1% to 78.6%, and the immich library (~198 GB)
+# projected to ~85-86%, Ceph's nearfull line.
+#
+# The projection: raw bytes of the restore spread evenly over all OSDs,
+# added to every OSD's usage now; the fullest result is the peak. Raw
+# bytes = sum of DataUpload totalBytes x the pool's raw/usable factor.
+# Even spread is exact for this cluster (three OSDs, one per host, every
+# pool spans all three); it would understate a skewed layout.
+OSD_MAX_PCT=${OSD_MAX_PCT:-80}
+PROMETHEUS_URL=${PROMETHEUS_URL:-http://kube-prometheus-stack-prometheus.monitoring.svc:9090}
+
+# Raw bytes written per usable byte, by StorageClass. Source of truth is
+# kubernetes/infrastructure/rook-ceph-cluster/ (storageclass.yaml,
+# cephfs-ci.yaml): cephfs-bulk is EC 2+1, cephfs-ci-data0 is size 2,
+# replicapool and cephfs-data0 are size 3. Any class not listed counts as
+# 3, the largest factor in the cluster.
+RAW_FACTORS='{"ceph-filesystem-bulk": 1.5, "ceph-filesystem-ci": 2}'
+
+# Prometheus instant query; echoes the first sample value.
+prom_scalar() {
+    curl -sf -m 30 -G --data-urlencode "query=$1" "${PROMETHEUS_URL}/api/v1/query" \
+        | jq -er '.data.result[0].value[1]'
+}
+
+# Echo "<raw_bytes> <now_pct> <peak_pct>" for a backup, or echo a reason
+# and return non-zero. Usage is read fresh per schedule, after the previous
+# schedule's teardown; deletion lag can only make it read higher, so it
+# errs on the side of skipping.
+preflight_capacity() {
+    local backup="$1" raw now peak
+    raw=$(kubectl get dataupload -n velero -l "velero.io/backup-name=${backup}" -o json \
+        | jq -er --argjson f "$RAW_FACTORS" '
+            [.items[] | {b: .status.progress.totalBytes,
+                         k: ($f[.spec.csiSnapshot.storageClass // ""] // 3)}]
+            | if any(.b == null) then error("no totalBytes") else map(.b * .k) | add | floor end') || {
+        echo "cannot size backup ${backup} (DataUpload without totalBytes?)"
+        return 1
+    }
+    now=$(prom_scalar 'max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes) * 100') \
+        && peak=$(prom_scalar "max((ceph_osd_stat_bytes_used + ${raw} / scalar(count(ceph_osd_stat_bytes))) / ceph_osd_stat_bytes) * 100") || {
+        echo "Prometheus query failed (${PROMETHEUS_URL})"
+        return 1
+    }
+    # NaN or an empty vector must not read as "below the gate".
+    if ! [[ "$now" =~ ^[0-9]+(\.[0-9]+)?$ && "$peak" =~ ^[0-9]+(\.[0-9]+)?$ ]]; then
+        echo "no usable OSD usage from Prometheus (now=${now} peak=${peak})"
+        return 1
+    fi
+    echo "$raw $now $peak"
+}
+
 # Force-delete Released PVs whose claimRef matches a pattern. The ceph-block
 # and ceph-filesystem StorageClasses ship with reclaimPolicy: Retain so that
 # production PVC deletions can be undone manually. The restore-test has the
@@ -428,6 +507,11 @@ for schedule in "${SCHEDULES[@]}"; do
    echo
    echo "=== service: ${src_ns} (schedule=${schedule}) ==="

+    if (( $(date +%s) >= SWEEP_DEADLINE )); then
+        skip "${src_ns}: restore (time budget)" "sweep budget of ${SWEEP_BUDGET_S}s used up"
+        continue
+    fi
+
    # Newest terminal Backup wins. PartiallyFailed is intentionally
    # included: Velero only restores DataUploads that completed, so
    # the partial source still gives a valid restore for the PVCs that
@@ -471,6 +555,23 @@ for schedule in "${SCHEDULES[@]}"; do
        continue
    fi

+    # Capacity gate. If the projection cannot be made, do not restore and
+    # count a FAIL: an unsized restore is the risk the gate exists for.
+    if ! cap=$(preflight_capacity "$backup"); then
+        check "${src_ns}: capacity pre-flight" bash -c "echo '$cap'; false"
+        continue
+    fi
+    read -r raw now peak <<<"$cap"
+    raw_gib=$(awk -v b="$raw" 'BEGIN { printf "%.0f", b / 2^30 }')
+    printf -v now '%.1f' "$now"
+    printf -v peak '%.1f' "$peak"
+    if awk -v p="$peak" -v m="$OSD_MAX_PCT" 'BEGIN { exit !(p > m) }'; then
+        skip "${src_ns}: restore (capacity gate)" \
+            "projected fullest OSD ${peak}% > ${OSD_MAX_PCT}% (now ${now}%, restore ~${raw_gib} GiB raw)"
+        continue
+    fi
+    log "${src_ns}: capacity ok, fullest OSD ${now}% -> projected ${peak}% (~${raw_gib} GiB raw)"
+
    # Pre-cleanup in case a prior run left residue.
    cleanup_service "$dst_ns" "" >/dev/null 2>&1 || true

@@ -492,8 +593,10 @@ spec:
    - persistentvolumes
 EOF

+    phase_timeout=$(( SWEEP_DEADLINE - $(date +%s) ))
+    (( phase_timeout > RESTORE_PHASE_TIMEOUT_S )) && phase_timeout=$RESTORE_PHASE_TIMEOUT_S
    check "${src_ns}: restore reaches phase=Completed" bash -c "
-        deadline=\$(( \$(date +%s) + ${RESTORE_PHASE_TIMEOUT_S} ))
+        deadline=\$(( \$(date +%s) + ${phase_timeout} ))
        last=''
        while [ \$(date +%s) -lt \$deadline ]; do
            p=\$(kubectl -n velero get restore.velero.io ${restore_name} \
@@ -509,7 +612,7 @@ EOF
            fi
            sleep 15
        done
-        echo 'TIMEOUT after ${RESTORE_PHASE_TIMEOUT_S}s'
+        echo 'TIMEOUT after ${phase_timeout}s'
        kubectl -n velero get restore.velero.io ${restore_name} -o yaml | tail -60
        exit 1
    "
@@ -523,6 +626,10 @@ EOF
        check "${src_ns}: at least one PVC was restored" false
    else
        for pvc in "${PVCS[@]}"; do
+            if (( $(date +%s) >= SWEEP_DEADLINE )); then
+                skip "${src_ns}: remaining PVC parity probes" "sweep time budget used up"
+                break
+            fi
            check "${src_ns}: PVC ${pvc} file count parity (±10% / ±${ABS_PARITY_SLACK})" \
                verify_pvc_file_count_parity "$src_ns" "$dst_ns" "$pvc"
        done
@@ -535,6 +642,14 @@ echo
 echo "=== Summary ==="
 echo "PASS: $PASS"
 echo "FAIL: $FAIL"
+echo "SKIP: $SKIP"
+# A skipped service has no restore evidence this month. Listed on every
+# run so it is not mistaken for a pass; it does not fail the Job.
+if [[ $SKIP -gt 0 ]]; then
+    echo
+    echo "Skipped (backup NOT restore-tested):"
+    printf '  - %s\n' "${SKIPS[@]}"
+fi
 if [[ $FAIL -gt 0 ]]; then
    echo
    echo "Failures:"
@@ -41,12 +41,12 @@
        "ghcr.io/immich-app/immich-server",
        "ghcr.io/matter-js/python-matter-server",
        "ghcr.io/music-assistant/server",
-        "ghcr.io/paperclipai/paperclip",
        "ghcr.io/paperless-ngx/paperless-ngx",
        "ghcr.io/usetrmnl/terminus",
        "ghcr.io/usetrmnl/trmnl-ha-amd64",
        "gitea.renner.uno/k.renner/shelly-manager",
        "gitea.renner.uno/k.renner/vsftpd",
+        "gitea.renner.uno/konairius/paperclip",
        "gitea.renner.uno/konairius/wiki",
        "gotenberg/gotenberg",
        "lscr.io/linuxserver/lidarr",