fix(crowdsec): request 50m CPU instead of the chart's 500m

Draining k8s-1 for the jupiter conversion left three pods unschedulable:
k8s-2 and k8s-3 both sat at 100% of requested CPU while actually using 21-23%.
Cluster-wide the gap is 19,680m requested against 7,957m used.

crowdsec is the clearest single contributor. The chart defaults reserve 500m
for each agent and for the LAPI; measured draw is 1-2m. As a DaemonSet the
agent alone parks half a core on every node. Nothing here had ever set
resources, so this was an inherited default rather than a decision.

Only the request changes. Limits stay at the chart default, so a log flood can
still burst to 500m; what shrinks is the reservation the scheduler subtracts
from every node whether or not the process uses it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-infrastructure.yml

@@ -13,7 +13,7 @@ when:
 
 steps:
   - name: validate-secrets
-    image: alpine:3.21
+    image: alpine:3.24.1
     environment:
       TF_VAR_proxmox_api_url:
         from_secret: proxmox_api_url
@@ -129,13 +129,20 @@ steps:
       - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
       - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
       - |
-        # Ensure Talos image exists on all Proxmox nodes via API
+        # Ensure Talos image exists on all Proxmox nodes via API.
+        #
+        # jupiter is NOT in this list: it runs Talos on bare metal and is no
+        # longer a Proxmox node, so the storage API returns nothing for it and
+        # the download-wait loop below would burn its full 60x10s before
+        # giving up. saturn and neptune still host k8s-2/k8s-3 as VMs and
+        # still need the nocloud image; bare-metal nodes boot the metal-amd64
+        # image from their own installed disk instead.
         TALOS_VERSION="v1.12.6"
         TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
         TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
         TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
         API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in jupiter saturn neptune; do
+        for node in saturn neptune; do
           EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
             -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
           if [ "$EXISTS" -gt 0 ]; then

kubernetes/RECOVERY.md

@@ -167,7 +167,7 @@ permanent recovery.
 
 ## CNPG restore-test (verify backups are restorable — non-destructive)
 
-The weekly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
+The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
 PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate
 code path). Run this drill periodically to prove the CNPG backups are actually
 restorable, **without touching the live clusters**. It restores into a throwaway

kubernetes/RUNBOOK-cold-rebuild.md

@@ -79,6 +79,30 @@ which also pre-seeds the Talos image onto every Proxmox node. Wait until the
 node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
 Step 2 — `bootstrap.sh` polls DNS but will not create the records.
 
+**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
+bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
+Before Step 2, boot jupiter from the bare-metal image below so it comes up in
+maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
+all three nodes identically from there, since `apply-config --insecure` does
+not care whether the target is a VM or bare metal.
+
+```
+https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
+```
+
+That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
+seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
+— without it the predictable interface names that `talos/patches/k8s-1.yaml`
+matches its VLAN parents against do not exist — and differs in three ways:
+
+- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
+- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
+- adds `siderolabs/thunderbolt`, unused so far and inert until
+  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
+  present only so enabling the MS-01 Thunderbolt mesh later does not require a
+  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
+  device attached to the machine once those modules are loaded.
+
 ## Step 2 — talos: form the cluster + get kubeconfig
 
 ```sh

kubernetes/apps/home-assistant/deployment.yaml

@@ -84,7 +84,7 @@ spec:
           env:
             - name: DB_URI
               valueFrom:
-                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
+                secretKeyRef: {name: ha-pg-app, key: fqdn-uri}
           volumeMounts:
             - name: config
               mountPath: /config
@@ -96,7 +96,7 @@ spec:
               memory: 32Mi
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml

@@ -176,3 +176,38 @@ data:
         friendly_name: Treppenhaus Haustür Schalter 1
       '0x001788010b00d011':
         friendly_name: Treppenhaus Haustür Schalter 2
+      # Ecowitt WS90 weather station. Joined after the ZHA-mirrored block above
+      # was written, hence listed apart from it.
+      #
+      # This device also carries a non-default ZCL reporting configuration
+      # (per-attribute min interval and reportable-change deadbands). That is
+      # device-side state, not expressible here — Z2M has no settings field for
+      # it — so it is documented, with restore commands, in
+      # zigbee2mqtt-ws90-reporting.md. It is lost whenever Z2M re-runs
+      # `configure` on the device, and a fresh PVC will not have it.
+      #
+      # The station transmits every 8.8s, as a burst of seven per-cluster ZCL
+      # frames; Z2M publishes its full cumulative state once per frame. debounce
+      # merges a burst into one publish carrying the last value per attribute,
+      # and must stay below the transmit interval — a window that never sees
+      # silence never flushes.
+      #
+      # Every few days the firmware zeroes its readings for ~8s, then skips a
+      # transmit cycle and recovers on the next. The zeros propagate attribute
+      # by attribute through a burst rather than arriving together, and they
+      # start inside an otherwise normal cycle — the last valid reading and the
+      # first zero are ~20ms apart.
+      #
+      # Merging cuts an episode from six published bad frames to one. It does
+      # NOT make that frame recognisable: it carries whatever subset had zeroed
+      # when the burst ended, which may be every field or only temperature, and
+      # a frame reading {"temperature":0,"humidity":92,"pressure":96.9} is not
+      # separable from a valid winter reading. Since the merge is last-wins and
+      # the zeros arrive at the tail, it also discards the valid reading that
+      # led the burst. This bounds the corruption; it does not filter it.
+      #
+      # Z2M reads this only at startup (requiresRestart), and this file is
+      # SEED-ONLY per the header — the running instance takes it from its PVC.
+      '0xc02cedfffe958f40':
+        friendly_name: Wetterstation
+        debounce: 2

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -158,7 +158,7 @@ spec:
           # Pinned tag. No Flux image-automation marker yet (unlike mosquitto) —
           # add an infrastructure/image-automation/zigbee2mqtt.yaml policy as a
           # follow-up if desired; pinned-and-manual is fine for the initial cutover.
-          image: ghcr.io/koenkk/zigbee2mqtt:2.12.1
+          image: ghcr.io/koenkk/zigbee2mqtt:2.13.0
           imagePullPolicy: IfNotPresent
           env:
             - name: TZ

kubernetes/apps/home-assistant/zigbee2mqtt-ws90-reporting.md

@@ -0,0 +1,149 @@
+# WS90 Zigbee reporting configuration
+
+Records the ZCL attribute-reporting configuration applied to the Wetterstation
+(Ecowitt WS90 powered by Shelly, IEEE `0xc02cedfffe958f40`) and how to restore
+it, because **this configuration lives on the device and in Z2M's
+`database.db`, not in git**. Zigbee2MQTT has no settings-schema field for
+reporting configuration, so there is nothing declarative to reconcile — Flux
+cannot restore it and a fresh PVC will not have it.
+
+## Applied configuration
+
+Endpoint 1, all intervals in seconds. `change` is in the attribute's raw ZCL
+units; the physical deadband is `change ÷ scale`.
+
+| attribute | cluster | min | max | change | deadband | converter default |
+| --- | --- | --- | --- | --- | --- | --- |
+| temperature | `msTemperatureMeasurement` (1026) | 5 | 3600 | 10 | 0.1 °C | 10 / 3600 / 100 |
+| humidity | `msRelativeHumidity` (1029) | 5 | 3600 | 50 | 0.5 % | 10 / 3600 / 100 |
+| pressure | `msPressureMeasurement` (1027) | 5 | 3600 | 1 | 1 hPa | 10 / 3600 / 50 |
+| illuminance | `msIlluminanceMeasurement` (1024) | 5 | 3600 | 5 | 5 lx | 10 / 3600 / 5 |
+| windSpeed | `shellyWS90Wind` (0xfc01/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
+| windDirection | `shellyWS90Wind` (0xfc01/4) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
+| gustSpeed | `shellyWS90Wind` (0xfc01/7) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
+| uvIndex | `shellyWS90UV` (0xfc02/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
+| rainStatus | `shellyWS90Rain` (0xfc03/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
+| precipitation | `shellyWS90Rain` (0xfc03/1) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
+
+`genPowerCfg.batteryVoltage` (cluster 1, attribute 33) is deliberately left at
+its hourly default.
+
+The three `shellyWS90*` clusters are manufacturer-specific and require
+`"options": {"manufacturerCode": 5264}` on every reconfiguration request.
+
+## Why min is 5 and not 10
+
+The station transmits every 8.8 s. `min` is a hold-off: a change-triggered
+report is held until at least `min` seconds after the previous one. With
+`min: 10` every report was therefore held past a transmit cycle and landed on
+the next, so attributes reported at best every 17.6 s and drifted out of phase
+with each other — the published stream showed no 8.8 s periodicity at all,
+just a smear across 2–8 s.
+
+At `min: 5` each attribute reports on its own transmit. The MQTT stream then
+carries one publish per transmit cycle, measured at a mean gap of 8.69 s.
+Total publish volume *fell* (~631/h to ~445/h) because each publish now
+carries a whole cycle instead of a fragment of one.
+
+Any value below 8.8 s achieves this. 5 leaves margin without approaching the
+cycle time.
+
+## Why the deadbands differ from the converter defaults
+
+The defaults come from the shared `modernExtend` helpers in
+zigbee-herdsman-converters (`m.temperature()`, `m.pressure()`, `m.humidity()`,
+`m.illuminance()`), which the WS90 definition in `shelly.ts` composes without
+overrides. They are anti-chatter values sized for battery-powered indoor
+sensors, where report volume costs coin-cell life and mesh airtime. This
+station is mains-and-solar and exists to resolve small changes, so the
+trade-off runs the other way.
+
+Each value above is matched to the resolution the device actually reports:
+temperature and pressure arrive in 0.1-unit steps, humidity in whole percent.
+
+Pressure is worth calling out. The converter default of `change: 50` with
+`scale: 10` is a 5 kPa (50 hPa) deadband, which real barometric pressure never
+traverses — so pressure only ever updated on the hourly `max` heartbeat, and
+`pressure_trend` and `weather_condition` were derived from that. The same
+library's older `reporting.ts` helper configures the identical ZCL attribute
+with `change: 5`, a 5 hPa deadband, so the two code paths disagree by 10× and
+the modernExtend value is the outlier.
+
+## When this is lost
+
+Anything that makes Z2M re-run `configure` on the device overwrites all of it
+with the converter defaults:
+
+- re-pairing or re-joining the device
+- an explicit `zigbee2mqtt/bridge/request/device/configure`
+- a converter update that changes the definition and triggers reconfiguration
+
+It also will not exist on a fresh PVC. Re-apply with the commands below.
+
+## Re-applying
+
+Published from the `discovery-guard` sidecar, which already holds the
+`zigbee2mqtt` broker credential (ACL `readwrite zigbee2mqtt/#`, which covers
+`bridge/request/#`) so no credential has to leave the cluster:
+
+Ten separate requests, one per attribute — written out literally rather than
+generated by a shell helper, because the nested quoting through `kubectl exec
+-- sh -c` is where this goes wrong. Each line below is in the exact form that
+was executed and verified.
+
+```sh
+POD=$(kubectl get pod -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt -o jsonpath='{.items[0].metadata.name}')
+```
+
+```sh
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msTemperatureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":10}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msRelativeHumidity\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":50}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msPressureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msIlluminanceMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":5}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windDirection\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"gustSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90UV\",\"attribute\":\"uvIndex\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"rainStatus\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
+
+kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"precipitation\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
+```
+
+These are idempotent — re-sending a request that matches the current
+configuration is a no-op on the device.
+
+The device must be awake and reachable; each request performs a real
+`ConfigureReporting` exchange and fails if it is not.
+
+## Verifying
+
+Read back what the device actually accepted — Z2M records the negotiated
+result, so this reflects the device, not the request:
+
+```sh
+kubectl exec -n home-assistant "$POD" -c zigbee2mqtt -- \
+  grep 0xc02cedfffe958f40 /app/data/database.db |
+  jq -r '.endpoints | to_entries[] | .value.configuredReportings[]?
+         | "cluster=\(.cluster) attr=\(.attrId) min=\(.minRepIntval) max=\(.maxRepIntval) change=\(.repChange)"'
+```
+
+All ten rows above should read `min=5 max=3600`. On the wire, the publish
+cadence on `zigbee2mqtt/Wetterstation` should sit at ~8.7 s and temperature
+should move in 0.1 °C steps; 1 °C steps mean the defaults are back.
+
+## Making it durable upstream
+
+The lasting fix is for the WS90 definition to pass explicit `reporting`
+overrides rather than inherit indoor-sensor defaults, which would fix every
+WS90 rather than this one. `m.temperature()` and the other helpers already
+accept a `reporting` argument. The pressure divergence between
+`modernExtend.ts` and `reporting.ts` is a separate upstream bug worth
+reporting on its own.

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -8,6 +8,7 @@ resources:
   - prometheusrule-cilium.yaml
   - prometheusrule-hetzner-backup.yaml
   - prometheusrule-loki.yaml
+  - prometheusrule-mariadb-backup.yaml
   - prometheusrule-netbird-availability.yaml
   - prometheusrule-netbird-backup.yaml
   - prometheusrule-netbird-ingress.yaml

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -25,13 +25,29 @@ spec:
           #
           # 6h: Ceph warns for self-clearing conditions. Only a warning that
           # does not clear is worth paging on.
+          #
+          # BLUESTORE_SLOW_OP_ALERT breaks that premise and is still NOT
+          # excluded (#557). Ceph holds the check for
+          # bluestore_slow_ops_warn_lifetime (24h) after the LAST slow op, so
+          # one multi-second stall outlives the 6h `for` and always pages,
+          # with latency back to normal long before anyone looks. Measured:
+          # both episodes in the 30d to 2026-08-23 ran ~26h and cleared
+          # untouched, the first ~24h after the last stall.
+          #
+          # Both began inside the Velero restore-test sweep — a ~5h sustained
+          # restore that outruns the SLC cache on these QLC NVMe. The sweep
+          # runs monthly, 08:00-14:00Z on the 1st.
+          #
+          # Expected for the hardware, but not excludable:
+          # this is also the earliest signal of a genuinely failing drive, and
+          # all three are at 80-87% endurance used (see cephcluster.yaml).
           expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
           for: 6h
           labels:
             severity: warning
           annotations:
             summary: "Ceph health check {{ $labels.name }} active for 6h"
-            description: "Ceph health check {{ $labels.name }} has been active for six hours, so it is not a self-clearing blip. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527), so this is a new condition. Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+            description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the monthly Velero restore-test window (08:00-14:00Z on the 1st), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
 
         - alert: CephPoolLostRedundancy
           # Names the pool, which POOL_NO_REDUNDANCY does not.

kubernetes/apps/monitoring-rules/prometheusrule-mariadb-backup.yaml

@@ -0,0 +1,51 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: mariadb-backup
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus selects on this label.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: mariadb-backup.rules
+      rules:
+        - alert: WordpressMariadbBackupStale
+          # The nightly 02:45 logical dump (kubernetes/apps/wordpress/mariadb.yaml)
+          # is the only transaction-consistent copy of www.renner.uno's database
+          # — Velero snapshots the PVC, which for a running DB is not a restore
+          # point you would trust.
+          #
+          # KubeJobFailed already covers a failed Job, but only while the Job
+          # object survives. On #550 it cleared when the failed Job was
+          # garbage-collected, not when a backup succeeded — so a failure that
+          # is GC'd before anyone looks leaves no signal at all.
+          # lastSuccessfulTime lives on the CronJob and outlives Job GC.
+          #
+          # 26h = one missed cycle plus 2h slack, matching NetbirdVpsBackupStale.
+          # It counts any Job owned by the CronJob, so a manual
+          # `kubectl create job --from=cronjob/...` re-run clears it too.
+          expr: |
+            time() - kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"} > 26 * 3600
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "WordPress MariaDB backup has not succeeded in 26h+"
+            description: "The wordpress-mariadb-backup CronJob's last successful run is over 26 hours old. Check the newest Job's pod logs in the wordpress namespace: `kubectl -n wordpress get jobs`. The 2026-08-19 failure was `The specified bucket does not exist.` against renner-uno-k8s-mariadb while the endpoint itself probed healthy, so a green HetznerBackupEndpointDown does not clear this."
+
+        - alert: WordpressMariadbBackupNeverRan
+          # The staleness expression above goes blind when the metric is
+          # absent, which is what happens once the operator recreates the
+          # CronJob. Without this guard "never backed up" is quieter than
+          # "backed up late".
+          expr: |
+            kube_cronjob_created{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
+            unless
+            kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
+          for: 6h
+          labels:
+            severity: warning
+          annotations:
+            summary: "WordPress MariaDB backup has no recorded successful run"
+            description: "The wordpress-mariadb-backup CronJob exists but has no lastSuccessfulTime: either it has not completed once since the Backup CR was recreated, or every run since has failed. Check `kubectl -n wordpress get jobs` and that the hetzner-s3-creds ExternalSecret is synced."

kubernetes/apps/woodpecker/agent.yaml

@@ -34,7 +34,7 @@ spec:
           # Image automation, range 3.x (see image-automation/woodpecker.yaml).
           # MUST stay version-matched with woodpecker-server (gRPC parity) — both
           # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
               value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
@@ -79,7 +79,7 @@ spec:
             # authoritative copy is the backup on Hetzner, and a lost probe just
             # means re-running the test -- so they are equally undeserving of
             # replication. Landing them unreplicated would mean a second size-1
-            # data pool inside `cephfs`, and restore-test runs WEEKLY against
+            # data pool inside `cephfs`, and restore-test runs MONTHLY against
             # CI's ~96 pipelines/day, so it is not worth the complexity.
             - name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
               value: ceph-filesystem-ci

kubernetes/apps/woodpecker/server.yaml

@@ -40,7 +40,7 @@ spec:
           # minor + patch auto-update within 3.x; only the 4.0 major is held for
           # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
           # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
           env:
             - name: WOODPECKER_HOST
               value: https://ci.renner.uno

kubernetes/e2e-test.sh

@@ -268,7 +268,7 @@ EOF
 
 # 120x15s = 30 min, not the old 60x5s = 5 min. node-agent has no
 # loadConcurrency override, so Velero's default of one data path per node
-# applies: while the Sunday restore-test sweep holds the slot, even this 1 GiB
+# applies: while the restore-test sweep holds the slot, even this 1 GiB
 # DataUpload sits in phase Prepared for well over 5 minutes. The old 300 s wait
 # then let the teardown below delete the source namespace and the Backup
 # mid-upload ("Cancel du under phase Prepared because it is being deleted"),

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -87,6 +87,16 @@ spec:
 
     lapi:
       replicas: 1
+      # The chart defaults request 500m. Measured steady-state draw is 2m, and
+      # the request is what the scheduler reserves, so the default parks half a
+      # core of unusable capacity per LAPI. That is affordable on three nodes
+      # and is not on two: with one node out for maintenance the remaining pair
+      # sit at 100% of requested CPU while actually using ~22%, and nothing new
+      # can be placed. The limit stays at the chart default so a log flood can
+      # still burst; only the reservation shrinks.
+      resources:
+        requests:
+          cpu: 50m
       podAnnotations:
         # Reloader restarts the LAPI when crowdsec-bouncer-key changes (it's an
         # env-var secret, so a restart is required to pick up a rotated key). See
@@ -122,6 +132,12 @@ spec:
               key: key
 
     agent:
+      # Same reasoning as lapi.resources above, multiplied by three: this is a
+      # DaemonSet, so the chart's 500m default reserves half a core on every
+      # node to run a process that measures 1-2m.
+      resources:
+        requests:
+          cpu: 50m
       # The agent parses the ingress controller's access logs (read from
       # the pod's stdout) to detect attacks. Repointed from the retired
       # ingress-nginx (decommissioned 2026-05-17) to Traefik. The

kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml

@@ -51,7 +51,15 @@ spec:
   startingDeadlineSeconds: 3600
   jobTemplate:
     spec:
-      backoffLimit: 1
+      # 4 (not the sibling rotators' 1-2) because this is the only DAILY
+      # rotator — every other CronJob here is quarterly, so it takes ~90x as
+      # many runs and is correspondingly more exposed to per-run transients.
+      # backoffLimit: 1 gave two attempts ~11 s apart, and on 2026-08-19 a
+      # sub-minute resolver transient took both out (Job
+      # claude-oauth-secret-sync-29785100, BackoffLimitExceeded), costing a
+      # day's sync. 4 spreads the attempts over ~150 s (10/20/40/80 s backoff),
+      # still well inside activeDeadlineSeconds.
+      backoffLimit: 4
       activeDeadlineSeconds: 300
       ttlSecondsAfterFinished: 259200 # 3 d
       template:

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -164,7 +164,15 @@ spec:
     useAllNodes: true
     useAllDevices: false
     devices:
-      - name: nvme0n1 # NVMe drive PCI-passed-through from each PVE node
+      # On k8s-2/k8s-3 this is the NVMe drive PCI-passed-through from saturn
+      # and neptune, and it is the only block device the guest can see. k8s-1
+      # is bare metal on jupiter and sees two: the OSD drive in MS-01 slot 1
+      # (PCI 01:00.0) and the Talos system disk in slot 2 (58:00.0). Slot 1
+      # enumerates first, so nvme0n1 is still the OSD drive there — but that
+      # is enumeration order, not a guarantee. The system disk is protected
+      # from the other direction instead: talos/patches/k8s-1.yaml pins the
+      # installer to it by serial, and Rook skips a disk carrying partitions.
+      - name: nvme0n1
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -87,15 +87,27 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
-  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
-  # measurable to drive wear — the entire reason for the size reduction — while
-  # losing it would destroy the whole filesystem, not just the objects on the
-  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
-  # trade; the wear saving comes from data0.
+  # TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
+  #
+  # The standing trade for this pool is size 3: it holds ~253 MiB against the
+  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
+  # the entire reason data0 dropped to 2 — while losing it would destroy the
+  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
+  # quarter of a gigabyte to keep that intact is the right trade in steady
+  # state.
+  #
+  # It does not hold while one of three hosts is out for the jupiter bare-metal
+  # conversion. size 3 on two remaining hosts leaves every PG undersized at two
+  # copies with min_size 2, so a single further OSD loss drops the pool below
+  # min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
+  # two copies are active+clean instead, and a further loss degrades rather
+  # than blocks — availability bought at the cost of accepting writes onto a
+  # single copy while degraded.
   metadataPool:
     replicated:
-      size: 3
+      size: 2
+    parameters:
+      min_size: "1"
   dataPools:
     - name: data0
       # size 2 / min_size 1 for the same reasons as replicapool above — see

kubernetes/infrastructure/velero/restore-test-cronjob.yaml

@@ -1,19 +1,30 @@
 ---
-# Weekly Velero restore-test CronJob.
+# Monthly Velero restore-test CronJob — 08:00 UTC on the 1st.
 #
-# Sunday 08:00 UTC — *after* Sunday's own daily backups have finished
-# (paperless 03:15 through apps 07:15, each normally ~1 min). The previous
-# 02:00 slot was chosen as "well before Monday's backups", which missed
-# that the daily schedules also fire on Sunday: a sweep starting at 02:00
-# and running ~5 h sat on top of the entire 03:15-07:15 backup window and
+# The sweep restores every discovered schedule, immich's ~180 GiB library
+# included. At `size: 2` that is ~360 GiB raw, ~120 GiB per OSD, against a
+# ~65 GB/day weekday baseline on three QLC NVMe at 81-87% endurance used —
+# ~47% of each drive's weekly write budget. Monthly keeps every restore path
+# exercised and cuts that share ~4x.
+#
+# 08:00 — *after* that day's own daily backups have finished (paperless
+# 03:15 through apps 07:15, each normally ~1 min). The previous 02:00 slot
+# was chosen as "well before the next day's backups", which missed that the
+# daily schedules fire on the same day: a sweep starting at 02:00 and
+# running ~5 h sat on top of the entire 03:15-07:15 backup window and
 # starved it of node-agent DataUpload concurrency. Observed effect on
 # 2026-07-19 and 2026-07-26 — audiobookshelf-daily took 2 h 18 m and
 # 2 h 33 m respectively against a ~1 min baseline, with the books
-# DataUpload not starting until 06:18. Both are Sundays; no other day
-# shows it.
+# DataUpload not starting until 06:18. Both are Sundays, the only day the
+# sweep ran then; no other day shows it.
 #
-# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of
-# Monday's 03:15.
+# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of the
+# next day's 03:15.
+#
+# Day-of-month rather than a fixed weekday: Kubernetes parses cron with
+# robfig/cron's standard parser, which has no `#` nth-weekday operator, and
+# restricting both day fields (`0 8 1-7 * 0`) ORs them — every Sunday plus
+# the 1st through 7th.
 #
 # Concurrency policy Forbid: the immich restore (~180 GiB, millions of
 # small files) alone can run multiple hours, so the full sweep is sized
@@ -29,7 +40,7 @@ metadata:
   name: restore-test
   namespace: velero
 spec:
-  schedule: "0 8 * * 0"
+  schedule: "0 8 1 * *"
   timeZone: Etc/UTC
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 2
@@ -39,7 +50,7 @@ spec:
     spec:
       backoffLimit: 0
       activeDeadlineSeconds: 21600  # 6 h hard cap (immich alone can take ~3 h)
-      ttlSecondsAfterFinished: 1209600  # 14 d — enough to inspect after the next weekly run
+      ttlSecondsAfterFinished: 3456000  # 40 d — enough to inspect after the next monthly run
       template:
         spec:
           serviceAccountName: restore-test

kubernetes/infrastructure/velero/restore-test-rbac.yaml

@@ -1,5 +1,5 @@
 ---
-# RBAC for the weekly restore-test CronJob.
+# RBAC for the monthly restore-test CronJob.
 #
 # Permissions required by restore-test.sh:
 #   - velero.io: read schedules/backups/BSL, create+delete restores

kubernetes/infrastructure/velero/restore-test.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Weekly verification that Velero PVC backups are actually restorable.
+# Monthly verification that Velero PVC backups are actually restorable.
 #
 # What this exercises (per service):
 #   1. Velero Restore reaches phase=Completed against yesterday's
@@ -53,7 +53,7 @@ check() {
 # and ceph-filesystem StorageClasses ship with reclaimPolicy: Retain so that
 # production PVC deletions can be undone manually. The restore-test has the
 # opposite need: every PVC it creates is throwaway and MUST not survive the
-# iteration. Without this, each weekly run leaks O(100 GiB) of CephFS
+# iteration. Without this, each run leaks O(100 GiB) of CephFS
 # subvolumes and the cluster eventually goes OSD_FULL (incident 2026-05-17).
 #
 # Why this still exists after ceph-*-ephemeral landed. Three categories of

kubernetes/talos/patches/k8s-1.yaml

@@ -1,62 +1,101 @@
+# k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
+# still Proxmox VMs. Every machine-specific selector below differs from theirs
+# for that reason: there are no virtio NICs, no PCI-passthrough layer, and both
+# NVMe drives are visible to Talos at once.
+#
+# Networking uses the v1.12 multi-document config rather than the deprecated
+# machine.network.interfaces. That is deliberate: the docs do not define what
+# happens when the same link is configured by both schemas, so this file keeps
+# every link in one schema instead of straddling them.
 machine:
   network:
+    # Shared control-plane VIP 10.2.40.60 is configured as a Layer2VIPConfig
+    # below rather than machine.network.interfaces[].vip. Exposed via UniFi DNS
+    # as k8s.renner.uno.
     hostname: k8s-1
-    # Shared control-plane VIP — etcd-elected leader hosts 10.2.40.60 as a
-    # secondary IP. Exposed via UniFi DNS as k8s.renner.uno.
-    #
-    # Two virtio NICs (mgmt + v30iot). `driver: virtio_net` would match
-    # both, so each is pinned by its PCI bus path:
-    #   - 0000:06:12.0 — original mgmt NIC (the only virtio before)
-    #   - 0000:06:13.0 — extra_network_bridges = ["v30iot"] in tofu
-    # Bus paths are consistent across k8s-{1,2,3}; verified live 2026-05-13.
-    # The X710 selector stays driver-based (i40e is unambiguous) and
-    # tolerates kernel renaming.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: LINK ONLY — no host addressing. The NIC is the macvlan
-      # parent for the v30iot pod attachments (matter/otbr/shelly-manager),
-      # whose children do their own DHCP on the L2; host addresses were
-      # never needed for that. Host DHCP (formerly dhcp + routeMetric 4096)
-      # installed a connected 10.2.30.0/24 route, so node replies to
-      # IoT-subnet clients bypassed the UDM and broke every UDM-DNATed
-      # flow (hairpin NAT: un-translated SYN-ACK -> client kernel RST,
-      # verified live 2026-07-23). The purpose the old comment claimed —
-      # Cilium L2 announce for an IoT LB pool — does not exist (no
-      # L2AnnouncementPolicy, no such pool). v6 RA addressing on this NIC
-      # is disabled via machine.sysctls for the same reason.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # Dedicated Ceph cluster network on the X710 SFP+ port (PCI passthrough,
-      # mapping x710-port1 in opentofu/vm_150_k8s_1.tf). Separate /29 on VLAN 49,
-      # MTU 9000, no default route — used by Rook for OSD↔OSD replication via
-      # CephCluster.spec.network.addressRanges.cluster.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.1/29
-        mtu: 9000
   sysctls:
-    # v30iot NIC carries no host addressing (see interfaces above); this
-    # disables v6 entirely (incl. link-local) so the host is unreachable on that L2.
-    #
-    # Keyed on the STABLE predictable name `ens19` (busPath 0000:06:13.0 =
-    # v30iot). Predictable naming is re-enabled via the Image Factory
-    # schematic (net.ifnames=1; see .woodpecker/deploy-infrastructure.yml),
-    # so `ens19` is topology-derived and does NOT shuffle across reboots —
-    # unlike the former `eth0/1/2` (driver-probe order) which renamed on a
-    # k8s-3 reboot and silently mis-targeted this sysctl onto mgmt. The four
-    # v30iot macvlan NADs key on the same `ens19`.
+    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
+    # the macvlan parent for the v30iot pod attachments
+    # (matter/otbr/shelly-manager/music-assistant), whose children do their own
+    # DHCP on that L2. Host addresses were never needed and are actively
+    # harmful: a connected 10.2.30.0/24 route makes node replies to IoT-subnet
+    # clients bypass the UDM, which breaks every UDM-DNATed flow (hairpin NAT:
+    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
+    # This disables v6 entirely on that link, including link-local.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Both NVMe drives are now ordinary block devices to Talos: the Kingston
+    # OM8PGP41024Q-A0 system disk and the Samsung PM983 that Rook claims as the
+    # Ceph OSD. Selecting the installer target by serial means it cannot land
+    # on the OSD drive regardless of which one the kernel enumerates first.
+    diskSelector:
+      serial: 50026B7382B48B86
   certSANs:
     - k8s-1.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: jupiter
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# Physical NICs are aliased by permanent MAC. Both X710 ports bind i40e, so the
+# driver selector k8s-2/k8s-3 use for their single passed-through port would
+# match two links here. MAC matching is unambiguous and survives udev renaming.
+apiVersion: v1alpha1
+kind: LinkAliasConfig
+name: lan0
+selector:
+  match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
+---
+apiVersion: v1alpha1
+kind: LinkAliasConfig
+name: ceph0
+selector:
+  match: glob("58:47:ca:74:70:28", mac(link.permanent_addr))
+---
+# lan0 is the switch trunk (X710 port 0). It carries no addressing of its own;
+# VLAN 40 and VLAN 30 ride on top. Proxmox's vlan-aware vmbr0 did this tagging
+# before, which is why the VM saw two already-untagged virtio NICs instead.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: lan0
+up: true
+---
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: lan0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# Named ens19 on purpose. The four v30iot macvlan NADs hard-code
+# `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
+# that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
+# serve both VM and bare-metal nodes for as long as the cluster is mixed.
+# Nothing else on this machine is called ens19: the physical NICs are
+# enp2s0f0np0, enp2s0f1np1, enp87s0 and enp89s0.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: lan0
+up: true
+---
+# Dedicated Ceph cluster network, VLAN 49 / 10.99.0.0/29, untagged on the wire
+# on X710 port 1. Used by Rook for OSD<->OSD replication via
+# CephCluster.spec.network.addressRanges.cluster. No default route.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph0
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.1/29

opentofu/vm_150_k8s_1.tf

@@ -1,51 +0,0 @@
-module "k8s_1" {
-  source = "./modules/proxmox_vm"
-
-  name        = "k8s-1.srv.renner.uno"
-  target_node = "jupiter"
-  vmid        = 150
-  memory      = 49152
-  cores       = 8
-
-  template_name = null
-  cloud_init    = false
-  full_clone    = false
-  bios          = "ovmf"
-  machine       = "q35"
-  boot          = "order=virtio0"
-  agent_enabled = true
-
-  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
-  # VLAN from this node (previously only k8s-edge had v30iot, so any
-  # IoT-pool LB lease landing here was unreachable from devices).
-  extra_network_bridges = ["v30iot"]
-
-  disks = [
-    {
-      storage = "local-zfs"
-      size    = "64G"
-      format  = "raw"
-      slot    = "virtio0"
-      file_id = local.talos_image
-    }
-  ]
-
-  hostpci = [
-    {
-      mapping = "osd-ssd"
-    },
-    {
-      mapping = "x710-port1"
-    }
-  ]
-
-  tags            = "kubernetes;talos;controlplane"
-  ssh_public_keys = local.common_ssh_keys
-
-  vm_state = "running"
-
-  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
-  # non-responsive Talos guest can't hang the destroy for 30 minutes.
-  stop_on_destroy = true
-}
-