fix: [alert] CephHealthError (rook-ceph) (#525 )

2026-08-19 16:46:25 +00:00
52 changed files with 290 additions and 1916 deletions
@@ -13,7 +13,7 @@ when:

 steps:
  - name: validate-secrets
-    image: alpine:3.24.1
+    image: alpine:3.21
    environment:
      TF_VAR_proxmox_api_url:
        from_secret: proxmox_api_url
@@ -129,20 +129,13 @@ steps:
      - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
      - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
      - |
-        # Ensure Talos image exists on all Proxmox nodes via API.
-        #
-        # Only nodes still running Proxmox belong in this list. jupiter and
-        # saturn run Talos on bare metal now, so the storage API returns
-        # nothing for them and the download-wait loop below would burn its full
-        # 60x10s per node before giving up. Bare-metal nodes boot the
-        # metal-amd64 image from their own installed disk instead, and the
-        # whole step goes away once neptune is converted too.
+        # Ensure Talos image exists on all Proxmox nodes via API
        TALOS_VERSION="v1.12.6"
        TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
        TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
        TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
        API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in neptune; do
+        for node in jupiter saturn neptune; do
          EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
            -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
          if [ "$EXISTS" -gt 0 ]; then
@@ -167,7 +167,7 @@ permanent recovery.

 ## CNPG restore-test (verify backups are restorable — non-destructive)

-The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
+The weekly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
 PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate
 code path). Run this drill periodically to prove the CNPG backups are actually
 restorable, **without touching the live clusters**. It restores into a throwaway
@@ -1,340 +0,0 @@
-# Runbook — converting a control-plane node from Proxmox VM to bare metal
-
-Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
-metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
-after doing jupiter on 2026-08-25; saturn and neptune remain.
-
-Everything here that reads like an over-specific warning is something that
-actually happened. The conversion itself is straightforward — the failures were
-all in state that outlived the VM.
-
-## What the node ends up as
-
-| | |
-|---|---|
-| System disk | Kingston OM8PGP41024Q-A0, selected by serial |
-| EPHEMERAL (`/var`) | the retired Crucial P3, selected by serial |
-| Ceph OSD | Samsung PM983, selected by `/dev/disk/by-id` |
-| Management | VLAN 40 tagged on X710 port 0 |
-| v30iot | VLAN 30 on the same port, named `ens19` |
-| Ceph cluster net | X710 port 1, `10.99.0.x/29`, MTU 9000 |
-
-## Before you start
-
- [ ] All three OSDs up, all mons in quorum, PGs `active+clean`. Do not start
-      from a degraded cluster; the window itself removes one of three.
- [ ] Cluster fits on two nodes. Check `kubectl describe node` allocation on the
-      two survivors. Take immich down (`kubernetes/apps/immich`, both
-      Deployments to `replicas: 0`) — it frees ~1000m against a shortfall of
-      roughly 200m and takes its image pulls off the remaining nodes.
- [ ] `cephfs-metadata` to size 2 / min_size 1 for the duration. At size 3 on
-      two hosts every PG is undersized with min_size 2, so one further OSD loss
-      blocks IO on every CephFS mount cluster-wide.
- [ ] Talos metal ISO on a USB stick. Schematic
-      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
-      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
-      Verify by comparing partition offsets and sizes against the ISO with
-      `fdisk -l` — blkid labels lie after a rewrite.
- [ ] Physical access. Secure Boot must be turned off (see below), and that is
-      the one step nothing can do remotely.
-
-## 1. Evacuate the OSD
-
-```sh
-ceph osd out <id>
-```
-
-Wait until no PGs are `backfilling` or `backfill_wait`. Do **not** wait for
-`ceph osd safe-to-destroy` — pools at size 3 keep the OSD in their acting set
-because three copies cannot be placed on two hosts, so it never returns clean.
-
-Then remove it. The node must already be out of `cephcluster.yaml`'s
-`storage.nodes` list, or the operator rebuilds the OSD within one reconcile and
-starts backfilling onto the drive you are about to pull:
-
-```sh
-kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0
-ceph osd purge <id> --yes-i-really-mean-it
-kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>
-```
-
-## 2. Drain, and leave the cluster
-
-```sh
-kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
-```
-
-Single-replica workloads with `minAvailable: 1` PDBs will refuse eviction — that
-is deliberate, they exist to surface the pod rather than let it be yanked. On
-this cluster that is `influxdb-influxdb2-0`, `woodpecker-server` and the
-CNPG primary. Move each one deliberately with `kubectl delete pod`; they
-reschedule onto the survivors and their RBD volumes follow.
-
-```sh
-talosctl -n <node> reset --graceful --wipe-mode system-disk
-kubectl delete node <node>
-```
-
-`--wipe-mode system-disk` and not the default `all`: the OSD drive is being
-pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
-etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
-the drain, which the rebuilt node would otherwise inherit.
-
-## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node
-
-Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the
-host leaves the Proxmox cluster: afterwards the provider cannot reach the node,
-the plan errors, and the only escape is `tofu state rm`, which is off the table.
-
-Point `proxmox_api_url` at a host that will *survive* the conversion, not the one
-being converted.
-
-Manual triggers are currently broken by the Woodpecker v3.18.0 regression
-(issue #582). Until that is resolved, run it locally:
-
-```sh
-AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=fsn1 \
-  tofu -chdir=opentofu init -backend-config=<file>
-… plan -target=module.k8s_N -var-file=<file> -out=<plan>
-… apply <plan>
-```
-
-The `AWS_*` environment variables are required *in addition to*
-`-backend-config`, or the S3 backend fails `HeadObject` with 403 even with
-credentials that can read the object fine via the aws CLI.
-
-## 2b. Proxmox quorum — matters from the second conversion onward
-
-Two nodes remain after jupiter, with `expected_votes: 2` and quorum 2. Removing
-one leaves the last node inquorate and `/etc/pve` read-only, which blocks the
-final VM destroy. On the surviving node:
-
-```sh
-pvecm expected 1
-```
-
-This is the official recovery. Proxmox documents it in the storage replication
-chapter (`/usr/share/pve-docs/chapter-pvesr.html`), verbatim:
-
-> If you have no quorum, we strongly advise to fix this first and make the node
-> operable again. Only if this is not possible at the moment, you may use the
-> following command to enforce quorum on the current node: `# pvecm expected 1`
-
-The caveat there is about restoring the missing node instead. Here the missing
-node is deliberately gone for good, so enforcing quorum is the correct branch
-rather than a workaround.
-
-Run it only once the node is actually alone — attempting it while both nodes are
-up fails with `CS_ERR_INVALID_PARAM`, since corosync will not set expected votes
-below the members it can currently see. Which is exactly why it becomes valid
-when one remains.
-
-Runtime only; it does not survive a reboot, so re-run it if the survivor
-restarts before its own conversion.
-
-## 3. Hardware
-
-Power the host off, then:
-
-1. Kingston boot drive out of slot 1, into slot 2.
-2. Crucial P3 out of slot 2 — it becomes EPHEMERAL, so put it in slot 3 rather
-   than setting it aside.
-3. Slot 1 switch to **U.2**. Set it before the drive goes in; there is nothing
-   to record afterwards, the installed drive type implies the position.
-4. PM983 onto the included 7mm M.2-to-U.2 adapter, into slot 1.
-
-Boot the host's existing PVE once before touching Talos and check the drive:
-
-```sh
-nvme smart-log /dev/nvmeXn1 | grep -iE 'percentage_used|media_errors|critical'
-nvme id-ns /dev/nvmeXn1 -H | grep -i 'in use'      # want 512b data, 0 metadata
-wipefs /dev/nvmeYn1                                 # Crucial: clear the old bluestore sig
-ip link set <ceph-port> up && ethtool <ceph-port> | grep -i 'link detected'
-```
-
-This is the last revert point. A bad drive here means the Crucial goes back and
-the VM is recreated from tofu with nothing lost. Once Talos writes to the
-Kingston, PVE on that host is gone.
-
-Relocating the boot drive is safe: `rpool` is referenced by NVMe EUI and the
-UEFI entries by GPT PARTUUID, neither of which encodes a slot.
-
-## 4. Secure Boot
-
-MS-01 ships with Secure Boot on, and Proxmox satisfies it with a
-Microsoft-signed shim. The Image Factory `metal-amd64.iso` is an unsigned UKI
-and the firmware rejects it outright. **Turn Secure Boot off in the BIOS.**
-
-A signed variant exists per schematic, but it is signed with SideroLabs' key
-which firmware does not trust by default, and the installed system would have to
-use the secureboot installer image too. That is a separate project.
-
-## 5. Install
-
-Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
-that lands on `192.168.8.x`, which is routable from the workstation.
-
-Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
-and `version` returns "API is not implemented in maintenance mode". You cannot
-inspect disks or links before applying. Verify serials and MACs from the host's
-PVE beforehand, in step 3.
-
-Build the config from the **live cluster's secrets**, not a fresh
-`talosctl gen config`, or the node forms a new cluster instead of joining:
-
-```sh
-talosctl -n <surviving-node> get mc v1alpha1 -o jsonpath='{.spec}' > cp.yaml
-talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
-talosctl gen config homelab https://k8s.renner.uno:6443 \
-  --with-secrets secrets.yaml --output-types controlplane \
-  --config-patch @talos/patches/cluster.yaml \
-  --config-patch @<cilium additive half> \
-  --config-patch @talos/patches/registry-mirror.yaml \
-  --config-patch @talos/patches/k8s-N.yaml -o install.yaml
-```
-
-Confirm `cluster.id` matches the running cluster before applying.
-
-Strip the generated `HostnameConfig` document — it collides with
-`machine.network.hostname` from the per-node patch. `bootstrap.sh` does the same.
-
-`cilium-cni.yaml` cannot be passed whole: its `$patch: delete` document fails
-against a base with no `inlineManifests` (issue #581). Use only the additive
-second document.
-
-Validate, then apply:
-
-```sh
-talosctl validate --config install.yaml --mode metal
-talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
-```
-
-## 6. The network config will strand the node if it is wrong
-
-Talos stops running default DHCP operators **for the entire machine** as soon as
-a single new-style network document exists — the gate is presence of a document,
-not success of resolution. If the configured path fails to materialise there is
-no fallback on any interface, and recovery needs a console and install media.
-
-Concretely: do not use `LinkAliasConfig` as a `VLANConfig` parent without
-verifying it. Aliases *are* supported (confirmed in v1.12.6 source and by a
-maintainer), but a selector matching zero links is skipped **silently**, and
-`permanent_addr` comes from an ethtool ioctl with no fallback — if the driver
-does not answer, the match is against an empty string and everything downstream
-that references the alias does nothing.
-
-Address links by predictable kernel name. To introduce aliases later, do it on a
-node that is already up, with `talosctl apply-config --mode=try --timeout=5m`,
-which rolls back on its own, and check `talosctl get linkaliasspec`.
-
-Interface names derive from PCI topology and are stable — but not immune.
-Adding a third NVMe to jupiter renumbered the buses and renamed the unused
-second I226 from `enp89s0` to `enp90s0`.
-
-## 7. EPHEMERAL placement is one-shot
-
-```yaml
-apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '<crucial serial>'
-  minSize: 50GiB
-  grow: true
-```
-
-Talos applies volume configuration **only when the volume does not yet exist**.
-Applied afterwards it is silently ignored, and relocating EPHEMERAL then needs a
-disk wipe and a fresh boot from install media. It must be in the first
-`apply-config`.
-
-Match by serial. `disk.transport == 'nvme' && !system_disk` matches the PM983 as
-well, and losing that coin flip formats the OSD drive.
-
-Verify afterwards:
-
-```sh
-talosctl -n <node> get volumestatus EPHEMERAL -o yaml | grep -E 'location|phase'
-talosctl -n <node> get disks
-```
-
-## 8. The mon will not come back on its own
-
-**This is the step that cost the most time on jupiter.** The rebuilt node has a
-new MAC and therefore a new DHCP lease, but Ceph has its mon pinned to the old
-address in three places. The mon cannot bind an address the host no longer has,
-so it crashloops; the operator then tries to restart a *healthy* mon as its next
-step, Ceph refuses because that would drop quorum, and it retries every 60
-seconds forever without ever reaching the storage reconcile. Nothing else
-progresses behind it.
-
-Fix in this order — the order matters, because doing it the other way round has
-the operator faithfully recreate the mon from the stale ConfigMap:
-
-```sh
-# 1. ConfigMap first
-kubectl -n rook-ceph edit cm rook-ceph-mon-endpoints
-#    remove the dead mon from `data` and from `mapping`
-
-# 2. then the monmap
-ceph mon remove <id>
-
-# 3. then the deployment — check it really carries the old address
-kubectl -n rook-ceph get deploy rook-ceph-mon-<id> -o yaml | grep public-addr
-kubectl -n rook-ceph delete deploy rook-ceph-mon-<id>
-```
-
-Rook then creates a **new** mon with the next free ID (jupiter's `c` came back
-as `d`), on the correct address. It does not repair the old one.
-
-Note mons use host ports under `network.provider: host`, so a replacement mon
-cannot schedule while the broken one's deployment still holds 3300/6789 — the
-canary sits `Pending` with "didn't have free ports" until the old deployment is
-deleted.
-
-## 9. Re-add the OSD
-
-Add the node back to `cephcluster.yaml` under `storage.nodes`, naming the PM983
-by `/dev/disk/by-id`:
-
-```yaml
- name: k8s-N
-  devices:
-    - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_<serial>
-```
-
-**Not** a bare device name. Talos enumerates these drives in a different order
-than Proxmox did — on jupiter `nvme0n1` is the Kingston *system* disk, and the
-PM983 is `nvme2n1`. Rook would probably decline a partitioned system disk, but
-that is luck rather than design.
-
-Get the path from `talosctl -n <node> get disk <dev> -o yaml`, under `symlinks`.
-
-## 10. Verify and restore
-
-```sh
-kubectl get nodes                                   # Ready
-talosctl -n <node> etcd members                     # 3 members
-ceph status                                         # 3 mons in quorum, 3 osds
-ceph osd tree                                       # OSD under the right zone/host
-ceph device ls                                      # PM983 backing the new OSD
-```
-
-Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
-1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
-raised to `high_recovery_ops` to speed the evacuation.
-
-The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
-`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
-
-## Known-stale things that will bite
-
- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
-  cold rebuild is broken independently of this work.
- Manual Woodpecker pipelines are broken (issue #582), so anything gated on
-  `PIPELINE == "..."` must be run another way.
- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.
@@ -79,30 +79,6 @@ which also pre-seeds the Talos image onto every Proxmox node. Wait until the
 node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
 Step 2 — `bootstrap.sh` polls DNS but will not create the records.

-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
-
-```
-https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
-```
-
-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
-
- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
-
 ## Step 2 — talos: form the cluster + get kubeconfig

 ```sh
@@ -121,9 +121,7 @@ spec:
          persistentVolumeClaim:
            claimName: audiobookshelf-books
        - name: le-ca
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service
@@ -43,44 +43,6 @@ spec:
    parameters:
      shared_buffers: "256MB"
      max_connections: "400"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
-      # and slower than the lz4/zstd this build also accepts). Compresses the
-      # full-page images that dominate WAL volume after each checkpoint.
-      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 3.2 GB/day here, most of it padding.
-      #
-      # 15min, NOT the 30min used elsewhere in this rollout. This cluster holds
-      # credentials, sessions and enrolment state, so its RPO is worth more than
-      # the difference: 30min would save a further 0.3 GB/day, which does not
-      # justify doubling the worst-case loss window on this data.
-      #
-      # It does carry a synchronous standby (instances: 2), so the realistic
-      # exposure is losing both instances, not one -- but the conservative value
-      # is cheap here and the aggressive one buys almost nothing.
-      archive_timeout: "15min"
    # Async replication makes CNPG switchover slow because the standby has
    # to chase missing WAL via barman-cloud and replay queued records
    # before it can promote. With one synchronous standby, the standby is
@@ -29,52 +29,6 @@ spec:
    parameters:
      shared_buffers: "128MB"
      max_connections: "50"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
-      # and slower than the lz4/zstd this build also accepts). Compresses the
-      # full-page images that dominate WAL volume after each checkpoint.
-      wal_compression: "zstd"
-      # THE dominant write source on this cluster, by three orders of magnitude.
-      #
-      # A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
-      # switch on a timer so the archive never lags reality by more than that
-      # interval -- and the switched-out segment is written and archived at its
-      # full 16 MiB regardless of how few bytes of real WAL it contains.
-      #
-      # CNPG defaults this to 5min, i.e. up to 288 segments/day = 4.6 GB/day.
-      # Measured here: this database is 21 MB and generates 4.5 MB/day of actual
-      # WAL, while its RBD volume takes 211 MB/hour. Essentially every byte
-      # written is padding -- roughly 1100x amplification.
-      #
-      # 30min cuts that to 48 segments/day. The cost is RPO: on total loss of
-      # the cluster, up to 30min of Grafana state is unrecoverable rather than
-      # 5min. Grafana's state here is dashboards, annotations and preferences,
-      # and the write rate says it changes ~4.5 MB/day -- so that window is
-      # nearly always empty anyway.
-      #
-      # Applied to gf-pg FIRST, alone, as a staged rollout: it is the lowest-risk
-      # cluster and has the cleanest signal, since its real WAL is small enough
-      # that any change in volume is unambiguously this parameter.
-      archive_timeout: "30min"
  resources:
    requests:
      cpu: 100m
@@ -66,52 +66,7 @@ Confirm the backup actually landed in the bucket rather than trusting the CR
 status — the WordPress backup failed nightly for three days against a bucket
 that was declared but never created (see `opentofu/object_storage.tf`).

-## Phase 2b — bulk statistics copy: DONE 2026-08-19
-
-Completed with **HA running** — the SQLite file was read through a read-only
-mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
-was needed for the bulk copy.
-
-```
-statistics_meta:        sqlite=1449       pg=1449       OK
-statistics:             sqlite=10799904   pg=10799904   OK   (153.9s)
-statistics_short_term:  sqlite=1969804    pg=1969804    OK   ( 27.0s)
-
-statistics span in Postgres: 2024-03-13 -> 2026-08-19
-entities with long-term history: 1449
-rows carrying sum (energy dashboard): 2627859
-database size: 2252 MB   (PVC is 20Gi)
-```
-
-`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
-`COPY` the planner has no statistics, and HA queries these tables constantly for
-history and energy views, so without it every view seq-scans 10.8M rows.
-
-Two schema details that cost a failed first attempt, recorded so the top-up does
-not repeat them:
-
- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
-  will not coerce; convert explicitly and preserve NULL.
- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
-  both bulk tables (superseded by the `_ts` variants), which is what makes a
-  tab-separated `COPY` safe — nothing in those tables needs escaping.
-
-The schema itself was created with HA's own models
-(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
-left EMPTY on purpose. HA's `_initialize_database` detects a new database by
-looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
-itself. Pre-stamping that row would skip HA's own initialisation path.
-
-### High-water marks for the cutover top-up
-
-Rows written after the bulk copy are not yet in Postgres. Resume from:
-
-```
-statistics             max_id=13323904    max_start_ts=1787162400.0
-statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
-```
-
-## Phase 2c — cutover (REMAINING)
+## Phase 2 — migrate the data (NOT in this PR)

 Do not start until phase 1 is verified.

@@ -135,18 +90,7 @@ HA outage, and buys a table that is discarded within 7 days anyway.
   SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
   ```
 5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
-   there, not here. The `recorder_db_url` secret it references is already
-   rendered into `secrets.yaml` by the `render-db-url` init container in
-   `deployment.yaml` (phase 2a), so this order is mandatory: **the init
-   container must be deployed first**, or HA refuses to start on a missing
-   secret.
-
-   The init container percent-encodes the credentials and validates the
-   resulting YAML rather than counting lines. That is not incidental — the CNPG
-   password is random, and a naive `"postgresql://user:pass@host/db"` string
-   makes `secrets.yaml` unparseable the moment the password contains a quote,
-   which takes out *every* secret HA loads, not just this one. Verified against
-   a deliberately hostile password before merge.
+   there, not here:
   ```yaml
   recorder:
     db_url: !secret recorder_db_url    # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
@@ -36,67 +36,9 @@ spec:
      # (sml2mqtt bridge owns the USB device on edge), so it no longer needs
      # the edge USB device and schedules on the general cluster pool. This
      # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
-      initContainers:
-        # Puts CNPG's connection URI into /config/secrets.yaml as
-        # `recorder_db_url`, for the recorder's move off SQLite
-        # (MIGRATION-recorder-postgres.md).
-        #
-        # HA has no way to read a db_url from the environment — `!env_var` was
-        # removed from its YAML loader — so `!secret` is the only mechanism, and
-        # secrets.yaml is PVC state that config-sync deliberately does not
-        # manage. Hence a small init container rather than a env: entry.
-        #
-        # The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
-        # percent-encodes it, so there is nothing to escape here: the value is
-        # restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
-        # An earlier version of this assembled the URI from user/password/dbname
-        # and needed percent-encoding plus YAML round-trip validation to be
-        # safe. That was solving a problem CNPG had already solved.
-        #
-        # Re-rendered on every pod start, so a CNPG password rotation takes
-        # effect rather than silently leaving a stale URI behind.
-        #
-        # INERT until the recorder is repointed: nothing reads this key until
-        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
-        # homeassistant repo. That ordering is required — HA refuses to start
-        # when a referenced secret is missing.
-        - name: render-db-url
-          image: busybox:1.36
-          command:
-            - sh
-            - -c
-            - |
-              set -eu
-              S=/config/secrets.yaml
-              T=/config/.secrets.render.tmp
-              [ -f "$S" ] || : > "$S"
-              # Filter-then-append, never rewrite in place: this file holds
-              # every other HA secret (influxdb_token and friends).
-              grep -v '^recorder_db_url:' "$S" > "$T" || true
-              printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
-              # Refuse to shrink the file. Cheap guard against the one failure
-              # that would actually hurt: losing the other keys.
-              [ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
-                echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
-              chmod 600 "$T"
-              mv "$T" "$S"
-              echo "recorder_db_url rendered"
-          env:
-            - name: DB_URI
-              valueFrom:
-                secretKeyRef: {name: ha-pg-app, key: fqdn-uri}
-          volumeMounts:
-            - name: config
-              mountPath: /config
-          resources:
-            requests:
-              cpu: 10m
-              memory: 16Mi
-            limits:
-              memory: 32Mi
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -279,13 +221,9 @@ spec:
        # Writable emptyDir for the nginx entrypoint's envsubst output
        # (/etc/nginx/conf.d/default.conf rendered from the .template).
        - name: proxy-conf-d
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
        # code-server's settings/extensions cache. Not on the config PVC
        # so installed extensions don't pollute HA's /config listing.
        # Trade-off: extensions reset across pod restarts.
        - name: vscode-state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
@@ -55,21 +55,7 @@ spec:
      owner: homeassistant
  postgresql:
    parameters:
-      # 512MB, not the 256MB this shipped with. Measured after 15h on Postgres:
-      # buffers_backend was 629,704 of 939,025 total page writes — 67% written
-      # by backend processes rather than the checkpointer or bgwriter. That
-      # ratio means the working set does not fit in shared_buffers, so backends
-      # evict dirty pages themselves and the same pages are written out
-      # repeatedly instead of coalescing in memory.
-      #
-      # The recorder touches a wide set of pages continuously (~30 state
-      # changes/sec across states, state_attributes and five indexes), which is
-      # exactly the workload that punishes a small buffer pool.
-      #
-      # Memory headroom checked before raising: the instance was using 383Mi of
-      # a 1Gi limit with 256MB of that being shared_buffers, so the limit goes
-      # to 1.5Gi to keep the same margin.
-      shared_buffers: "512MB"
+      shared_buffers: "256MB"
      max_connections: "50"
      # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
      # entire reason for the migration. Defaults would give a correct database
@@ -85,70 +71,12 @@ spec:
      # Compress full-page images in WAL. Recorder writes are many small row
      # inserts scattered across pages, so full-page writes after each
      # checkpoint dominate WAL volume; compression cuts that directly.
-      #
-      # Full-page images really are ~62% of WAL here: 61.9% measured as a
-      # steady-state delta over one checkpoint cycle, against 62.4% lifetime.
-      #
-      # But switching pglz -> zstd changed NOTHING. Measured by
-      # pg_current_wal_lsn advance over 20min windows: 4911 and 4976 MB/day
-      # before, 5002 MB/day after.
-      #
-      # The reason is that `wal_compression: "on"` was never "no compression" --
-      # it is pglz, which was already compressing these page images. zstd only
-      # changes the ratio, and on recorder pages that margin is too small to
-      # measure. The original framing of this change (that the clusters "were
-      # not getting compression") was wrong; they were.
-      #
-      # Note the 62% figure is an upper bound: it is computed as
-      # wal_fpi * 8192 / wal_bytes, which assumes uncompressed 8 KiB pages. With
-      # compression active the true byte share is lower, which is consistent
-      # with there being little headroom left for a better algorithm.
-      #
-      # Kept because zstd is not worse than pglz and costs nothing. It is simply
-      # not a lever. The real one on this cluster was archive_timeout, below.
-      #
-      # zstd, not "on": the boolean spelling resolves to pglz, the original
-      # 2016-era algorithm, which is both slower and weaker than the two
-      # alternatives PostgreSQL 16 accepts here (lz4, zstd). zstd gives the
-      # best ratio on 8 KiB page images, and the CPU cost lands on a cluster
-      # that is IO-bound on QLC NVMe, not CPU-bound.
-      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
+      wal_compression: "on"
      # Fewer checkpoints => fewer full-page writes, since a page is only
-      # fully-logged on its first modification after a checkpoint.
-      #
-      # That reasoning did not show up in WAL volume: 15min -> 30min moved the
-      # WAL rate not at all (4911/4976 -> 5002 MB/day). The likely reason is
-      # that this workload is append-mostly -- a page is dirtied while it fills
-      # and then not touched again, so its full-page image is paid once
-      # regardless of how often checkpoints run. Interval only helps pages that
-      # are re-dirtied across many cycles, which is not what a recorder does.
-      #
-      # Kept, but NOT because WAL improved. checkpoint_timeout also governs
-      # data-file writes at checkpoint, and those are unmeasured here: this
-      # volume takes ~11 GB/day against 4.9 GB/day of WAL, so most of its
-      # traffic is outside what the WAL measurement covers. Revisit with a
-      # data-file measurement before treating this value as justified.
-      #
-      # Recovery cost is bounded by how much WAL a single interval produces,
-      # not by the interval itself: at the measured rate 30min is well under
-      # 200MB of replay, i.e. seconds. max_wal_size stays 2GB precisely so the
-      # timeout stays the binding trigger -- if size were the binding one, the
-      # interval change would do nothing.
-      checkpoint_timeout: "30min"
+      # fully-logged on its first modification after a checkpoint. 15min with a
+      # 2GB ceiling keeps recovery time sane while cutting repeat full-page
+      # images for the hot rows the recorder rewrites constantly.
+      checkpoint_timeout: "15min"
      max_wal_size: "2GB"
      min_wal_size: "512MB"
      # full_page_writes is deliberately LEFT ON (the default). RBD can tear a
@@ -159,4 +87,4 @@ spec:
      cpu: 200m
      memory: 512Mi
    limits:
-      memory: 1536Mi
+      memory: 1Gi
@@ -313,9 +313,7 @@ spec:
          configMap:
            name: mosquitto-config
        - name: auth
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
        - name: tls
          secret:
            secretName: mosquitto-tls
@@ -176,38 +176,3 @@ data:
        friendly_name: Treppenhaus Haustür Schalter 1
      '0x001788010b00d011':
        friendly_name: Treppenhaus Haustür Schalter 2
-      # Ecowitt WS90 weather station. Joined after the ZHA-mirrored block above
-      # was written, hence listed apart from it.
-      #
-      # This device also carries a non-default ZCL reporting configuration
-      # (per-attribute min interval and reportable-change deadbands). That is
-      # device-side state, not expressible here — Z2M has no settings field for
-      # it — so it is documented, with restore commands, in
-      # zigbee2mqtt-ws90-reporting.md. It is lost whenever Z2M re-runs
-      # `configure` on the device, and a fresh PVC will not have it.
-      #
-      # The station transmits every 8.8s, as a burst of seven per-cluster ZCL
-      # frames; Z2M publishes its full cumulative state once per frame. debounce
-      # merges a burst into one publish carrying the last value per attribute,
-      # and must stay below the transmit interval — a window that never sees
-      # silence never flushes.
-      #
-      # Every few days the firmware zeroes its readings for ~8s, then skips a
-      # transmit cycle and recovers on the next. The zeros propagate attribute
-      # by attribute through a burst rather than arriving together, and they
-      # start inside an otherwise normal cycle — the last valid reading and the
-      # first zero are ~20ms apart.
-      #
-      # Merging cuts an episode from six published bad frames to one. It does
-      # NOT make that frame recognisable: it carries whatever subset had zeroed
-      # when the burst ended, which may be every field or only temperature, and
-      # a frame reading {"temperature":0,"humidity":92,"pressure":96.9} is not
-      # separable from a valid winter reading. Since the merge is last-wins and
-      # the zeros arrive at the tail, it also discards the valid reading that
-      # led the burst. This bounds the corruption; it does not filter it.
-      #
-      # Z2M reads this only at startup (requiresRestart), and this file is
-      # SEED-ONLY per the header — the running instance takes it from its PVC.
-      '0xc02cedfffe958f40':
-        friendly_name: Wetterstation
-        debounce: 2
@@ -158,7 +158,7 @@ spec:
          # Pinned tag. No Flux image-automation marker yet (unlike mosquitto) —
          # add an infrastructure/image-automation/zigbee2mqtt.yaml policy as a
          # follow-up if desired; pinned-and-manual is fine for the initial cutover.
-          image: ghcr.io/koenkk/zigbee2mqtt:2.13.0
+          image: ghcr.io/koenkk/zigbee2mqtt:2.12.1
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
@@ -344,13 +344,9 @@ spec:
          configMap:
            name: zigbee2mqtt-interview-watch-script
        - name: interview-watch-state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
        - name: discovery-guard-script
          configMap:
            name: zigbee2mqtt-discovery-guard-script
        - name: discovery-guard-state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
@@ -1,149 +0,0 @@
-# WS90 Zigbee reporting configuration
-
-Records the ZCL attribute-reporting configuration applied to the Wetterstation
-(Ecowitt WS90 powered by Shelly, IEEE `0xc02cedfffe958f40`) and how to restore
-it, because **this configuration lives on the device and in Z2M's
-`database.db`, not in git**. Zigbee2MQTT has no settings-schema field for
-reporting configuration, so there is nothing declarative to reconcile — Flux
-cannot restore it and a fresh PVC will not have it.
-
-## Applied configuration
-
-Endpoint 1, all intervals in seconds. `change` is in the attribute's raw ZCL
-units; the physical deadband is `change ÷ scale`.
-
-| attribute | cluster | min | max | change | deadband | converter default |
-| --- | --- | --- | --- | --- | --- | --- |
-| temperature | `msTemperatureMeasurement` (1026) | 5 | 3600 | 10 | 0.1 °C | 10 / 3600 / 100 |
-| humidity | `msRelativeHumidity` (1029) | 5 | 3600 | 50 | 0.5 % | 10 / 3600 / 100 |
-| pressure | `msPressureMeasurement` (1027) | 5 | 3600 | 1 | 1 hPa | 10 / 3600 / 50 |
-| illuminance | `msIlluminanceMeasurement` (1024) | 5 | 3600 | 5 | 5 lx | 10 / 3600 / 5 |
-| windSpeed | `shellyWS90Wind` (0xfc01/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| windDirection | `shellyWS90Wind` (0xfc01/4) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| gustSpeed | `shellyWS90Wind` (0xfc01/7) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| uvIndex | `shellyWS90UV` (0xfc02/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| rainStatus | `shellyWS90Rain` (0xfc03/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| precipitation | `shellyWS90Rain` (0xfc03/1) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-
-`genPowerCfg.batteryVoltage` (cluster 1, attribute 33) is deliberately left at
-its hourly default.
-
-The three `shellyWS90*` clusters are manufacturer-specific and require
-`"options": {"manufacturerCode": 5264}` on every reconfiguration request.
-
-## Why min is 5 and not 10
-
-The station transmits every 8.8 s. `min` is a hold-off: a change-triggered
-report is held until at least `min` seconds after the previous one. With
-`min: 10` every report was therefore held past a transmit cycle and landed on
-the next, so attributes reported at best every 17.6 s and drifted out of phase
-with each other — the published stream showed no 8.8 s periodicity at all,
-just a smear across 2–8 s.
-
-At `min: 5` each attribute reports on its own transmit. The MQTT stream then
-carries one publish per transmit cycle, measured at a mean gap of 8.69 s.
-Total publish volume *fell* (~631/h to ~445/h) because each publish now
-carries a whole cycle instead of a fragment of one.
-
-Any value below 8.8 s achieves this. 5 leaves margin without approaching the
-cycle time.
-
-## Why the deadbands differ from the converter defaults
-
-The defaults come from the shared `modernExtend` helpers in
-zigbee-herdsman-converters (`m.temperature()`, `m.pressure()`, `m.humidity()`,
-`m.illuminance()`), which the WS90 definition in `shelly.ts` composes without
-overrides. They are anti-chatter values sized for battery-powered indoor
-sensors, where report volume costs coin-cell life and mesh airtime. This
-station is mains-and-solar and exists to resolve small changes, so the
-trade-off runs the other way.
-
-Each value above is matched to the resolution the device actually reports:
-temperature and pressure arrive in 0.1-unit steps, humidity in whole percent.
-
-Pressure is worth calling out. The converter default of `change: 50` with
-`scale: 10` is a 5 kPa (50 hPa) deadband, which real barometric pressure never
-traverses — so pressure only ever updated on the hourly `max` heartbeat, and
-`pressure_trend` and `weather_condition` were derived from that. The same
-library's older `reporting.ts` helper configures the identical ZCL attribute
-with `change: 5`, a 5 hPa deadband, so the two code paths disagree by 10× and
-the modernExtend value is the outlier.
-
-## When this is lost
-
-Anything that makes Z2M re-run `configure` on the device overwrites all of it
-with the converter defaults:
-
- re-pairing or re-joining the device
- an explicit `zigbee2mqtt/bridge/request/device/configure`
- a converter update that changes the definition and triggers reconfiguration
-
-It also will not exist on a fresh PVC. Re-apply with the commands below.
-
-## Re-applying
-
-Published from the `discovery-guard` sidecar, which already holds the
-`zigbee2mqtt` broker credential (ACL `readwrite zigbee2mqtt/#`, which covers
-`bridge/request/#`) so no credential has to leave the cluster:
-
-Ten separate requests, one per attribute — written out literally rather than
-generated by a shell helper, because the nested quoting through `kubectl exec
-- sh -c` is where this goes wrong. Each line below is in the exact form that
-was executed and verified.
-
-```sh
-POD=$(kubectl get pod -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt -o jsonpath='{.items[0].metadata.name}')
-```
-
-```sh
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msTemperatureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":10}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msRelativeHumidity\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":50}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msPressureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msIlluminanceMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":5}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windDirection\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"gustSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90UV\",\"attribute\":\"uvIndex\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"rainStatus\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"precipitation\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-```
-
-These are idempotent — re-sending a request that matches the current
-configuration is a no-op on the device.
-
-The device must be awake and reachable; each request performs a real
-`ConfigureReporting` exchange and fails if it is not.
-
-## Verifying
-
-Read back what the device actually accepted — Z2M records the negotiated
-result, so this reflects the device, not the request:
-
-```sh
-kubectl exec -n home-assistant "$POD" -c zigbee2mqtt -- \
-  grep 0xc02cedfffe958f40 /app/data/database.db |
-  jq -r '.endpoints | to_entries[] | .value.configuredReportings[]?
-         | "cluster=\(.cluster) attr=\(.attrId) min=\(.minRepIntval) max=\(.maxRepIntval) change=\(.repChange)"'
-```
-
-All ten rows above should read `min=5 max=3600`. On the wire, the publish
-cadence on `zigbee2mqtt/Wetterstation` should sit at ~8.7 s and temperature
-should move in 0.1 °C steps; 1 °C steps mean the defaults are back.
-
-## Making it durable upstream
-
-The lasting fix is for the WS90 definition to pass explicit `reporting`
-overrides rather than inherit indoor-sensor defaults, which would fix every
-WS90 rather than this one. `m.temperature()` and the other helpers already
-accept a `reporting` argument. The pressure divergence between
-`modernExtend.ts` and `reporting.ts` is a separate upstream bug worth
-reporting on its own.
@@ -42,43 +42,7 @@ spec:
    parameters:
      shared_buffers: "512MB"
      max_wal_size: "2GB"
-      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
-      # and slower than the lz4/zstd this build also accepts). Compresses the
-      # full-page images that dominate WAL volume after each checkpoint.
-      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
+      wal_compression: "on"
  resources:
    requests:
      cpu: 500m
@@ -6,13 +6,6 @@ metadata:
  labels:
    app.kubernetes.io/name: immich-machine-learning
 spec:
-  # Scale this and immich-server to 0 for the duration of any single-node
-  # maintenance — cluster CPU requests exceed what two nodes can hold, and this
-  # pair frees ~1000m against a shortfall of roughly 200m. Stopping them is the
-  # right lever rather than lowering their requests: 500m reserved against a
-  # 2m idle draw looks like waste, but inference bursts hard and the request is
-  # honest about what it needs when someone uploads photos.
-  # See RUNBOOK-baremetal-conversion.md.
  replicas: 1
  strategy:
    type: RollingUpdate
@@ -27,7 +27,7 @@ spec:
        # Pinned to 7.4-alpine to match the LXC 225 compose exactly.
        # Any redis major-version bump is a separate migration.
        - name: redis
-          image: redis:7.4.11-alpine # {"$imagepolicy": "flux-system:redis-immich"}
+          image: redis:7.4.10-alpine # {"$imagepolicy": "flux-system:redis-immich"}
          args:
            - redis-server
            - --save
@@ -36,9 +36,6 @@ metadata:
  labels:
    app.kubernetes.io/name: immich-server
 spec:
-  # Scaled to 0 alongside immich-machine-learning during single-node
-  # maintenance; see the note there. The CNPG Postgres cluster stays running,
-  # so recovery is just these two replica counts.
  replicas: 1
  strategy:
    type: Recreate
@@ -150,9 +147,7 @@ spec:
              - key: immich-config.json
                path: immich-config.json
        - name: le-ca
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service
@@ -8,16 +8,11 @@ resources:
  - prometheusrule-cilium.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
-  - prometheusrule-mariadb-backup.yaml
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
  - prometheusrule-velero.yaml
  - prometheusrule-woodpecker-rotation.yaml
-  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
-  # alerted if that stopped — HetznerBackupEndpointDown only probes the S3
-  # endpoint, which proves reachability, not that anything was written.
-  - prometheusrule-cnpg-backup.yaml
  - alloy-servicemonitor.yaml
  - cnpg-podmonitors.yaml
  - home-assistant-servicemonitor.yaml
@@ -12,63 +12,46 @@ spec:
    - name: ceph.rules
      rules:
        - alert: CephHealthWarn
-          # Per-check rather than ceph_health_status, so one permanently-active
-          # check cannot mask every other Ceph warning.
-          #
-          # AUTH_INSECURE_* is excluded: it cannot clear until the nodes run a
-          # kernel supporting aes256k over the CSI mount path (#527).
-          #
-          # POOL_NO_REDUNDANCY is in scope. An unreplicated pool makes
-          # `ceph osd ok-to-stop` refuse for every OSD, so it reports blocked
-          # maintenance and not just data risk — do not exclude it for a pool
-          # that is unreplicated deliberately.
-          #
-          # 6h: Ceph warns for self-clearing conditions. Only a warning that
-          # does not clear is worth paging on.
-          #
-          # BLUESTORE_SLOW_OP_ALERT breaks that premise and is still NOT
-          # excluded (#557). Ceph holds the check for
-          # bluestore_slow_ops_warn_lifetime (24h) after the LAST slow op, so
-          # one multi-second stall outlives the 6h `for` and always pages,
-          # with latency back to normal long before anyone looks. Measured:
-          # both episodes in the 30d to 2026-08-23 ran ~26h and cleared
-          # untouched, the first ~24h after the last stall.
-          #
-          # Both began inside the Velero restore-test sweep — a ~5h sustained
-          # restore that outruns the SLC cache on these QLC NVMe. The sweep
-          # runs monthly, 08:00-14:00Z on the 1st.
-          #
-          # Expected for the hardware, but not excludable:
-          # this is also the earliest signal of a genuinely failing drive, and
-          # all three are at 80-87% endurance used (see cephcluster.yaml).
-          expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
+          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
+          # 6h is deliberately long. Ceph warns for self-clearing conditions —
+          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
+          # below its disk threshold — and paging on those trains people to
+          # ignore the alert. What matters is a warning that does not clear.
+          expr: ceph_health_status == 1
          for: 6h
          labels:
            severity: warning
          annotations:
-            summary: "Ceph health check {{ $labels.name }} active for 6h"
-            description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the monthly Velero restore-test window (08:00-14:00Z on the 1st), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
-
-        - alert: CephPoolLostRedundancy
-          # Names the pool, which POOL_NO_REDUNDANCY does not.
-          #
-          # There is no pool-size metric, but bytes_used/stored IS the
-          # replication factor. The >1e6 floor drops pools too small to measure
-          # meaningfully, empty ones included.
-          #
-          # No pool is exempt.
-          expr: |
-            ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
-          for: 30m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
-            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
+            summary: "Ceph has been HEALTH_WARN for 6h"
+            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."

        - alert: CephHealthError
-          # HEALTH_ERR means data availability or redundancy is at risk.
-          expr: ceph_health_status == 2
+          # HEALTH_ERR used to mean "data availability or redundancy is at
+          # risk". Ceph 19.2.6 broke that assumption: it added the
+          # AUTH_INSECURE_* cephx checks, two of them ERR-severity, which flag
+          # the legacy `aes` key type every keyring in this cluster still uses.
+          # That is a security-posture finding, not a durability one — see
+          # CephAuthInsecureKeys below, which is what actually tracks it.
+          #
+          # The problem with leaving `ceph_health_status == 2` alone is not
+          # noise, it is blindness: while the auth checks hold the cluster at
+          # HEALTH_ERR, a genuine PG_AVAILABILITY or OSD_FULL is
+          # indistinguishable from them, because the aggregate is already
+          # saturated. So page on the ERR checks that are *not* auth ones.
+          #
+          # Written as "unless every active ERR check is an AUTH_INSECURE_ one"
+          # rather than a direct match so that it fails open: if
+          # ceph_health_detail goes missing or Ceph renames a check, both counts
+          # come back empty, the `unless` removes nothing, and this still pages.
+          # The `> 0` matters — the mgr exports all 31 known checks, the
+          # inactive ones at 0, so an unfiltered count would compare constants.
+          expr: |
+            ceph_health_status == 2
+              unless on (cluster) (
+                count by (cluster) (ceph_health_detail{severity="HEALTH_ERR"} > 0)
+                  ==
+                count by (cluster) (ceph_health_detail{severity="HEALTH_ERR", name=~"AUTH_INSECURE_.*"} > 0)
+              )
          for: 5m
          labels:
            severity: critical
@@ -76,6 +59,22 @@ spec:
            summary: "Ceph is HEALTH_ERR"
            description: "The Ceph cluster is in HEALTH_ERR — availability or redundancy is at risk. Inspect immediately with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and `ceph osd df tree`."

+        - alert: CephAuthInsecureKeys
+          # Carries the condition CephHealthError deliberately stops paging for,
+          # so excluding it there narrows the alert rather than silencing it.
+          # Warning, not critical: nothing is at risk of being lost, but this
+          # does not self-clear either — it needs a planned cephx key rotation
+          # across client.admin, the CSI provisioner/node keys and the RGW key.
+          # Rotating those carelessly breaks RBD/CephFS provisioning
+          # cluster-wide, so this stays a human decision, not a 3am one.
+          expr: count(ceph_health_detail{name=~"AUTH_INSECURE_.*"} > 0) > 0
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Ceph cephx keys use insecure key types"
+            description: "Ceph reports {{ $value }} active AUTH_INSECURE_* health checks — cephx entities still using the legacy `aes` key type, flagged since the 19.2.6 upgrade. No data is at risk. Enumerate with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and plan rotation against the 19.2.6 release notes; durable settings belong in `spec.cephConfig` in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml."
+
        - alert: CephMonDiskLow
          # Named explicitly rather than left to CephHealthWarn, because the
          # mons sit on the node's /var alongside container storage: this
@@ -1,78 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: cnpg-backup
-  namespace: monitoring
-  labels:
-    # kube-prometheus-stack's Prometheus resource selects on this label;
-    # without it the rule is created but never loaded into Prometheus.
-    release: kube-prometheus-stack
-spec:
-  groups:
-    - name: cnpg-backup.rules
-      rules:
-        # WHY THIS FILE EXISTS
-        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
-        # stopped. The only related rule was HetznerBackupEndpointDown, which
-        # probes the S3 endpoint — it proves the bucket is reachable, not that
-        # anything was written to it. Same failure shape as the WordPress backup
-        # that failed nightly for three days against a bucket that was declared
-        # in tofu but never applied: the status surface looked fine because
-        # nothing checked the thing that mattered.
-
-        - alert: CNPGWalArchiveBacklog
-          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
-          # to be archived. Steady state is 0 across every cluster (verified
-          # 2026-08-19). A sustained backlog means the archiver is failing:
-          # point-in-time recovery is silently broken from that moment, and
-          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
-          #
-          # This is the load-bearing alert of the two. It depends only on the
-          # instance-level collector, so unlike the backup-age metrics below it
-          # is unaffected by the missing Backup CRs (#522).
-          #
-          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
-          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
-          # 80 MiB of un-archived WAL, well below anything that threatens the
-          # volume but far above idle noise.
-          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
-          for: 30m
-          labels:
-            severity: warning
-          annotations:
-            summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
-
-        - alert: CNPGBackupFailed
-          # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
-          # when no failure is recorded. Guard on > 0 so the epoch-zero default
-          # is not read as "failed in 1970".
-          expr: |
-            cnpg_collector_last_failed_backup_timestamp > 0
-            and
-            (time() - cnpg_collector_last_failed_backup_timestamp) < 86400
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
-
-        # DELIBERATELY ABSENT: a backup-age alert, which is the one you would
-        # most want here.
-        #
-        # The natural expression is
-        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
-        # and it CANNOT be used today: that metric reads exactly 0 on every
-        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
-        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
-        # successful backup anywhere, because the Backup CRs it derives that
-        # from are being deleted within a minute of completing — see #522.
-        #
-        # Backups themselves DO work; that was verified independently by
-        # listing the bucket with barman-cloud-backup-list and restoring a WAL
-        # segment back out of S3. The gap is in the record, not the data.
-        #
-        # Adding the rule now would fire permanently for all seven clusters and
-        # be silenced within a day, which is worse than not having it. Add it as
-        # part of resolving #522, once the metric reflects reality.
@@ -1,51 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: mariadb-backup
-  namespace: monitoring
-  labels:
-    # kube-prometheus-stack's Prometheus selects on this label.
-    release: kube-prometheus-stack
-spec:
-  groups:
-    - name: mariadb-backup.rules
-      rules:
-        - alert: WordpressMariadbBackupStale
-          # The nightly 02:45 logical dump (kubernetes/apps/wordpress/mariadb.yaml)
-          # is the only transaction-consistent copy of www.renner.uno's database
-          # — Velero snapshots the PVC, which for a running DB is not a restore
-          # point you would trust.
-          #
-          # KubeJobFailed already covers a failed Job, but only while the Job
-          # object survives. On #550 it cleared when the failed Job was
-          # garbage-collected, not when a backup succeeded — so a failure that
-          # is GC'd before anyone looks leaves no signal at all.
-          # lastSuccessfulTime lives on the CronJob and outlives Job GC.
-          #
-          # 26h = one missed cycle plus 2h slack, matching NetbirdVpsBackupStale.
-          # It counts any Job owned by the CronJob, so a manual
-          # `kubectl create job --from=cronjob/...` re-run clears it too.
-          expr: |
-            time() - kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"} > 26 * 3600
-          for: 30m
-          labels:
-            severity: warning
-          annotations:
-            summary: "WordPress MariaDB backup has not succeeded in 26h+"
-            description: "The wordpress-mariadb-backup CronJob's last successful run is over 26 hours old. Check the newest Job's pod logs in the wordpress namespace: `kubectl -n wordpress get jobs`. The 2026-08-19 failure was `The specified bucket does not exist.` against renner-uno-k8s-mariadb while the endpoint itself probed healthy, so a green HetznerBackupEndpointDown does not clear this."
-
-        - alert: WordpressMariadbBackupNeverRan
-          # The staleness expression above goes blind when the metric is
-          # absent, which is what happens once the operator recreates the
-          # CronJob. Without this guard "never backed up" is quieter than
-          # "backed up late".
-          expr: |
-            kube_cronjob_created{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
-            unless
-            kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
-          for: 6h
-          labels:
-            severity: warning
-          annotations:
-            summary: "WordPress MariaDB backup has no recorded successful run"
-            description: "The wordpress-mariadb-backup CronJob exists but has no lastSuccessfulTime: either it has not completed once since the Backup CR was recreated, or every run since has failed. Check `kubectl -n wordpress get jobs` and that the hetzner-s3-creds ExternalSecret is synced."
@@ -402,52 +402,6 @@ spec:
        limits:
          memory: 256Mi

-    # Drop the apiserver's high-cardinality latency histograms.
-    #
-    # Measured 2026-08-20: Prometheus held 298,236 active series, of which
-    # ~108,000 — over a third — were apiserver/etcd `_bucket` histograms. Each
-    # histogram fans out to one series per bucket x verb x resource x scope, so
-    # a single metric like apiserver_request_duration_seconds_bucket was 23,810
-    # series on its own. For comparison, every Home Assistant metric combined is
-    # ~4,100.
-    #
-    # That cardinality is the dominant driver of Prometheus write volume
-    # (~11 GB/day of TSDB + WAL, doubled again by Ceph replication), and none of
-    # it is observable value here: per-bucket apiserver latency quantiles are for
-    # clusters with SLAs and dozens of nodes, not three.
-    #
-    # ONLY metrics with ZERO rule dependencies are dropped. Verified against the
-    # live rule set before writing this — each metric checked individually for
-    # references in alert and recording rules:
-    #
-    #   apiserver_request_duration_seconds_bucket        rules=0   23,810 series
-    #   apiserver_request_body_size_bytes_bucket         rules=0   23,200
-    #   etcd_request_duration_seconds_bucket             rules=0   15,840
-    #   apiserver_watch_list_duration_seconds_bucket     rules=0   11,538
-    #   apiserver_watch_cache_read_wait_seconds_bucket   rules=0    8,232
-    #   apiserver_response_sizes_bucket                  rules=0    7,592
-    #   apiserver_watch_events_sizes_bucket              rules=0    5,220
-    #
-    # apiserver_request_sli_duration_seconds_bucket is DELIBERATELY KEPT: 23
-    # recording rules depend on it, including the apiserver availability SLO and
-    # the KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently
-    # break that alerting, which is the failure mode this comment exists to
-    # prevent if someone later "tidies up" this regex.
-    #
-    # Only the `_bucket` series are dropped. The matching `_count` and `_sum`
-    # remain, so request rates and mean latencies still work — what is lost is
-    # per-bucket quantiles.
-    #
-    # etcd_* appears here even though kubeEtcd is disabled below: these are the
-    # apiserver's own etcd-client metrics, exposed on the apiserver endpoint, not
-    # a separate etcd scrape.
-    kubeApiServer:
-      serviceMonitor:
-        metricRelabelings:
-          - action: drop
-            sourceLabels: [__name__]
-            regex: (apiserver_request_duration_seconds_bucket|apiserver_request_body_size_bytes_bucket|etcd_request_duration_seconds_bucket|apiserver_watch_list_duration_seconds_bucket|apiserver_watch_cache_read_wait_seconds_bucket|apiserver_response_sizes_bucket|apiserver_watch_events_sizes_bucket)
-
    # Disable etcd/scheduler/controller-manager scraping (Talos doesn't expose them by default)
    kubeEtcd:
      enabled: false
@@ -64,9 +64,7 @@ spec:
          secret:
            secretName: pve-exporter-config
        - name: tmp
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service
@@ -1,97 +0,0 @@
-# paperless-data: CephFS → RBD
-
-## Why
-
-`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
-Deployment mounts it**. Verified 2026-08-20:
-
-```
-$ pods mounting paperless-data
-paperless-68cb8b8744-7hjb8      <- one, and only one
-
-paperless-data     ReadWriteMany  ceph-filesystem
-paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-```
-
-Its contents are single-writer state — the 50 MB Whoosh index,
-`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
-`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
-
-It is on CephFS by default, not by requirement.
-
-## What it costs
-
-Every access goes through the MDS. Measured with **zero CI pipelines running**:
-
-```
-paperless-data client   request_load_avg 22   (all other sessions 0-1)
-MDS over 180s           4,052 client requests  = 22.5/s
-                        2,718 journal events   = 15/s
-                            2 dir commits      <- almost none: not file writes
-```
-
-So it is cap/inode churn from repeated access, not data being written. That is
-most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
-pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
-
-`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
-filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
-depends on it. Cutting the writes at source is correct; cutting the replication
-is not.
-
-## Sequence
-
-Staged deliberately: the PVC lands first and empty, so merging it changes
-nothing. The copy and the repoint are separate.
-
-### Phase 1 — provision (this PR)
-
-`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
-nothing. Inert.
-
-Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
-
-### Phase 2 — copy
-
-The Whoosh index must come across intact. Paperless *can* rebuild it
-(`document_index reindex`), but that reprocesses every document, so copying is
-both faster and less disruptive.
-
-1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
-   and the Whoosh index are both live-written, and copying them under an active
-   writer gives a torn read.
-2. Run a one-off pod that mounts BOTH PVCs and copies:
-   ```
-   cp -a /src/. /dst/
-   ```
-   The RWO target pins the pod to one node; the RWX source will follow.
-3. Verify file count and index size match:
-   ```
-   find /src -type f | wc -l   ==   find /dst -type f | wc -l
-   du -sh /src/index           ==   du -sh /dst/index
-   ```
-
-### Phase 3 — repoint
-
-Change the Deployment's `data` volume to `claimName: paperless-data-block`,
-merge, let it roll.
-
-Verify afterwards:
- paperless starts, search returns results (proves the Whoosh index survived)
- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
-  client
- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
-
-**Do not delete `paperless-data` until search has been confirmed working.** It is
-the rollback, and it is `prune: disabled` so Flux will not remove it.
-
-## Expected effect
-
-~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
-replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
-worth doing, but it is a correctness fix (a single-writer volume does not belong
-on a shared filesystem) more than a large win.
-
-It also removes 22 req/sec of MDS load, which benefits the workloads that
-genuinely need CephFS — Woodpecker CI workspaces share the same MDS.
@@ -34,42 +34,6 @@ spec:
    parameters:
      shared_buffers: "256MB"
      max_connections: "200"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
-      # and slower than the lz4/zstd this build also accepts). Compresses the
-      # full-page images that dominate WAL volume after each checkpoint.
-      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.1 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if both instances
-      # are lost. The documents themselves live on the media PVC, not in
-      # Postgres -- this database holds their metadata.
-      archive_timeout: "30min"
    # See ak-pg cluster.yaml for the rationale on synchronous replication.
    synchronous:
      method: any
@@ -208,25 +208,9 @@ spec:
            seccompProfile:
              type: RuntimeDefault
      volumes:
-        # paperless-data-block (RWO, ceph-block), NOT paperless-data
-        # (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
-        # and its contents are single-writer state — the Whoosh index,
-        # classification model, celerybeat schedule, logs. Being on CephFS meant
-        # every access went through the MDS: 22 requests/sec sustained with zero
-        # CI running, ~15 journal events/sec against 2 directory commits per
-        # 180s, i.e. cap/inode churn rather than file writes. That was most of
-        # the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
-        #
-        # `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
-        # those from another node, so they genuinely need RWX.
-        #
-        # paperless-data is retained (prune: disabled) as the rollback. Do not
-        # delete it until search has been confirmed working — that is what
-        # proves the Whoosh index survived the copy, not merely that paperless
-        # starts. Migration record: MIGRATION-data-to-block.md.
        - name: data
          persistentVolumeClaim:
-            claimName: paperless-data-block
+            claimName: paperless-data
        - name: media
          persistentVolumeClaim:
            claimName: paperless-media
@@ -234,9 +218,7 @@ spec:
          persistentVolumeClaim:
            claimName: paperless-consume
        - name: bridge-cert
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service
@@ -18,45 +18,6 @@ spec:
    requests:
      storage: 10Gi
 ---
-# Replacement for paperless-data, on RBD instead of CephFS.
-#
-# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
-# pod ever mounts it — the paperless Deployment itself. Its contents are all
-# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
-# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
-# and `media` genuinely need RWX, because the FTP scanner writes to those from a
-# different node.
-#
-# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
-# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
-# and the MDS logged ~15 journal events/sec against only 2 directory commits per
-# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
-# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
-# pool, so ~21-24 GB/day actually hits the drives.
-#
-# cephfs-metadata deliberately STAYS at size 3 (see storageclass.yaml): losing
-# the metadata pool destroys the whole filesystem, not just the objects on one
-# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
-# is the right fix; reducing the replication is not.
-#
-# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
-# and the deployment repoint are a separate, staged step — see
-# MIGRATION-data-to-block.md. Merging this alone is inert.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-data-block
-  namespace: paperless
-  annotations:
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block
-  resources:
-    requests:
-      storage: 10Gi
---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -25,7 +25,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: redis
-          image: redis:8.10.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
+          image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
          args:
            - redis-server
            - --save
@@ -32,44 +32,7 @@ spec:
    parameters:
      shared_buffers: "128MB"
      max_connections: "50"
-      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
-      # and slower than the lz4/zstd this build also accepts). Compresses the
-      # full-page images that dominate WAL volume after each checkpoint.
-      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
+      wal_compression: "on"
  resources:
    requests:
      cpu: 100m
@@ -362,22 +362,6 @@ spec:
            name: trmnl-renderer-schedule
        - name: tmp
          emptyDir:
-            # RAM-backed, NOT disk. Without `medium: Memory` an emptyDir lands
-            # on the node's filesystem, and this one was the single largest
-            # writer in the cluster: 249 GB over 7.3 days = ~34 GB/day onto
-            # /dev/vda5, more than every Ceph OSD and every database combined
-            # (measured 2026-08-20 from the container's cgroup io.stat).
-            #
-            # The cause is cadence, not volume. schedules.template.json renders
-            # every 60 s, so Chrome is launched 1440 times a day and each launch
-            # creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-*
-            # directory. At rest /tmp holds under 8 MB; it is pure churn for
-            # data that never outlives the render.
-            #
-            # tmpfs pages count against the container's memory limit, which is
-            # why the cap stays at 512Mi: measured usage is 129Mi against a 2Gi
-            # limit, so even the full cap leaves ample headroom.
-            medium: Memory
            sizeLimit: 512Mi
 ---
 apiVersion: v1
@@ -34,7 +34,7 @@ spec:
          # Image automation, range 3.x (see image-automation/woodpecker.yaml).
          # MUST stay version-matched with woodpecker-server (gRPC parity) — both
          # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
          env:
            - name: WOODPECKER_SERVER
              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
@@ -50,39 +50,12 @@ spec:
            # clone + subsequent steps share the workspace PVC and the k8s backend
            # can schedule them on different nodes -> an RWO volume then fails to
            # mount on the second node. CephFS (RWX) lets the workspace follow
-            # the pipeline across nodes. Reclaim MUST be Delete: workspaces are
-            # throwaway, and a Retain class leaked one Released PV per finished
-            # pipeline (issue #262 follow-up).
-            #
-            # ceph-filesystem-ci, not ceph-filesystem-ephemeral: a dedicated
-            # CephFS whose pools are BOTH size 1 (infrastructure/
-            # rook-ceph-cluster/cephfs-ci.yaml). The shared filesystem
-            # replicates data0 2x and metadata 3x, which is the correct trade
-            # for real data and pure waste for a git clone that is deleted
-            # minutes later.
-            #
-            # Measured 2026-08-21, polling both pools every 15s (a 5-minute
-            # interval misses an entire pipeline -- PVC created, used and
-            # deleted inside ~2 minutes):
-            #   CI active  1.5min: data0 143.2 MB, metadata 17.2 MB
-            #   CI idle   16.0min: data0   0.0 MB, metadata 11.0 MB
-            # data0 write volume is 100% CI. Per pipeline this moves the drive
-            # cost from 71.6x2 + 8.2x3 = ~168 MB to ~80 MB.
-            #
-            # ceph-filesystem-ephemeral is NOT repointed, and the reason is a
-            # technical constraint rather than a safety one. Its other consumer
-            # is velero/restore-test.sh, whose probe PVCs are snapshot CLONES:
-            # `ceph fs subvolume snapshot clone` operates within one filesystem,
-            # so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
-            #
-            # Those probes are just as throwaway as a CI workspace -- the
-            # authoritative copy is the backup on Hetzner, and a lost probe just
-            # means re-running the test -- so they are equally undeserving of
-            # replication. Landing them unreplicated would mean a second size-1
-            # data pool inside `cephfs`, and restore-test runs MONTHLY against
-            # CI's ~96 pipelines/day, so it is not worth the complexity.
+            # the pipeline across nodes. The -ephemeral variant (reclaim Delete,
+            # rook-ceph-cluster/storageclass.yaml) is REQUIRED here: workspaces
+            # are throwaway, and the Retain class leaked one Released PV per
+            # finished pipeline (issue #262 follow-up).
            - name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
-              value: ceph-filesystem-ci
+              value: ceph-filesystem-ephemeral
            - name: WOODPECKER_BACKEND_K8S_STORAGE_RWX
              value: "true"
            # MUST be a JSON object literal — the backend json.Unmarshal's this
@@ -56,44 +56,6 @@ spec:
    parameters:
      shared_buffers: "128MB"
      max_connections: "50"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
-      # and slower than the lz4/zstd this build also accepts). Compresses the
-      # full-page images that dominate WAL volume after each checkpoint.
-      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
  resources:
    requests:
      cpu: 100m
@@ -40,7 +40,7 @@ spec:
          # minor + patch auto-update within 3.x; only the 4.0 major is held for
          # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
          # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
          env:
            - name: WOODPECKER_HOST
              value: https://ci.renner.uno
@@ -268,7 +268,7 @@ EOF

 # 120x15s = 30 min, not the old 60x5s = 5 min. node-agent has no
 # loadConcurrency override, so Velero's default of one data path per node
-# applies: while the restore-test sweep holds the slot, even this 1 GiB
+# applies: while the Sunday restore-test sweep holds the slot, even this 1 GiB
 # DataUpload sits in phase Prepared for well over 5 minutes. The old 300 s wait
 # then let the teardown below delete the source namespace and the Backup
 # mid-upload ("Cancel du under phase Prepared because it is being deleted"),
@@ -917,27 +917,11 @@ subjects:
  name: plugin-barman-cloud
  namespace: database
 ---
-# stringData, NOT base64 `data`. An image reference is not a secret, and
-# encoding it here had two real costs: no reviewer sees a version change in a
-# diff, and no $imagepolicy marker can be attached to a base64 blob. The result
-# was that automation advanced the controller to v0.14.0 while this sidecar sat
-# at v0.12.0 — a two-minor skew in the component that performs every base backup
-# and WAL archive, invisible until someone read the pod spec (#535).
-#
-# The Secret KIND is kept because the controller reads it via secretKeyRef
-# (deployment env SIDECAR_IMAGE); changing it to a ConfigMap would need a matching
-# edit to that reference and diverge further from upstream. stringData is
-# accepted identically by the API server and serialised into `data` on write.
-#
-# The name carries a kustomize hash suffix from upstream's secretGenerator. It is
-# static here, so editing the value does NOT change the name and the deployment's
-# secretKeyRef stays valid.
-#
-# IF YOU RE-IMPORT THE UPSTREAM MANIFEST: upstream ships this as base64 `data`
-# with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
-stringData:
-  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
+data:
+  SIDECAR_IMAGE: |
+    Z2hjci5pby9jbG91ZG5hdGl2ZS1wZy9wbHVnaW4tYmFybWFuLWNsb3VkLXNpZGVjYXI6dj
+    AuMTIuMA==
 kind: Secret
 metadata:
  name: plugin-barman-cloud-kg5c2795f7
@@ -87,16 +87,6 @@ spec:

    lapi:
      replicas: 1
-      # The chart defaults request 500m. Measured steady-state draw is 2m, and
-      # the request is what the scheduler reserves, so the default parks half a
-      # core of unusable capacity per LAPI. That is affordable on three nodes
-      # and is not on two: with one node out for maintenance the remaining pair
-      # sit at 100% of requested CPU while actually using ~22%, and nothing new
-      # can be placed. The limit stays at the chart default so a log flood can
-      # still burst; only the reservation shrinks.
-      resources:
-        requests:
-          cpu: 50m
      podAnnotations:
        # Reloader restarts the LAPI when crowdsec-bouncer-key changes (it's an
        # env-var secret, so a restart is required to pick up a rotated key). See
@@ -132,12 +122,6 @@ spec:
              key: key

    agent:
-      # Same reasoning as lapi.resources above, multiplied by three: this is a
-      # DaemonSet, so the chart's 500m default reserves half a core on every
-      # node to run a process that measures 1-2m.
-      resources:
-        requests:
-          cpu: 50m
      # The agent parses the ingress controller's access logs (read from
      # the pod's stdout) to detect attacks. Repointed from the retired
      # ingress-nginx (decommissioned 2026-05-17) to Traefik. The
@@ -28,40 +28,3 @@ spec:
  policy:
    semver:
      range: "0.x"
---
-# The SIDECAR image, tracked separately from the controller above.
-#
-# These are two DIFFERENT images that must move together:
-#   plugin-barman-cloud          the controller, in the database namespace
-#   plugin-barman-cloud-sidecar  injected into every Postgres pod
-#
-# Only the controller was tracked, so automation walked it from v0.12.0 to
-# v0.14.0 while the sidecar stayed at v0.12.0 — a two-minor skew in the
-# component that performs every base backup and WAL archive (#535). The skew was
-# invisible because the sidecar tag was base64-encoded inside a Secret, where no
-# $imagepolicy marker can reach it and no reviewer would see it in a diff.
-apiVersion: image.toolkit.fluxcd.io/v1
-kind: ImageRepository
-metadata:
-  name: cnpg-barman-plugin-sidecar
-  namespace: flux-system
-spec:
-  image: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar
-  interval: 1h
---
-# Same filter and range as the controller policy deliberately: the two images
-# are released together and must not be allowed to drift apart again.
-apiVersion: image.toolkit.fluxcd.io/v1
-kind: ImagePolicy
-metadata:
-  name: cnpg-barman-plugin-sidecar
-  namespace: flux-system
-spec:
-  imageRepositoryRef:
-    name: cnpg-barman-plugin-sidecar
-  filterTags:
-    pattern: '^v(?P<ver>\d+\.\d+\.\d+)$'
-    extract: '$ver'
-  policy:
-    semver:
-      range: "0.x"
@@ -156,6 +156,4 @@ spec:
        # each pod enrolls its own peer and a restart re-enrolls cleanly
        # (the old peer is reaped).
        - name: state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
@@ -51,15 +51,7 @@ spec:
  startingDeadlineSeconds: 3600
  jobTemplate:
    spec:
-      # 4 (not the sibling rotators' 1-2) because this is the only DAILY
-      # rotator — every other CronJob here is quarterly, so it takes ~90x as
-      # many runs and is correspondingly more exposed to per-run transients.
-      # backoffLimit: 1 gave two attempts ~11 s apart, and on 2026-08-19 a
-      # sub-minute resolver transient took both out (Job
-      # claude-oauth-secret-sync-29785100, BackoffLimitExceeded), costing a
-      # day's sync. 4 spreads the attempts over ~150 s (10/20/40/80 s backoff),
-      # still well inside activeDeadlineSeconds.
-      backoffLimit: 4
+      backoffLimit: 1
      activeDeadlineSeconds: 300
      ttlSecondsAfterFinished: 259200 # 3 d
      template:
@@ -17,17 +17,6 @@ spec:
    modules:
      - name: pg_autoscaler
        enabled: true
-  # Rotates daemon keys to aes256k for CVE-2025-30156 (#527). Opt-in; disabled
-  # by default. Increment keyGeneration to rotate again.
-  #
-  # Daemon keys only — aes256k over the CSI mount path needs kernel 7.0+ and
-  # these nodes run 6.18, so CSI stays on aes and the client-key checks stay
-  # raised. keyType is unset so Rook picks it.
-  security:
-    cephx:
-      daemon:
-        keyRotationPolicy: KeyGeneration
-        keyGeneration: 2
  # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
  # Data Movement provisions a backupPVC per source PVC by cloning the
  # VolumeSnapshot, and a 4-thread default stalls migrate-immich's
@@ -160,41 +149,11 @@ spec:
        - "10.2.40.0/24"
      cluster:
        - "10.99.0.0/29"
-  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
-  # taken out of the storage fleet by removing it here. Under useAllNodes the
-  # operator re-provisions an OSD on any node whose device still carries a
-  # BlueStore signature, which makes purging an OSD ahead of pulling its drive
-  # impossible: the OSD is recreated within a reconcile and starts backfilling
-  # onto the drive being replaced.
-  #
-  # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see, so a bare device
-  # name is unambiguous there.
-  #
-  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
-  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
-  # enumerates this machine's drives in a different order than Proxmox did, and
-  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
-  #
-  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
-  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   nvme2n1  S437NX0M400426    PM983     — this OSD
  storage:
-    useAllNodes: false
+    useAllNodes: true
    useAllDevices: false
-    nodes:
-      # k8s-2 is absent while saturn's OSD drive is swapped and the node is
-      # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
-      # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare
-      # metal sees three NVMe drives and the bare name is not the OSD.
-      - name: k8s-3
-        devices:
-          - name: nvme0n1
-      - name: k8s-1
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
+    devices:
+      - name: nvme0n1 # NVMe drive PCI-passed-through from each PVE node
    config:
      osdsPerDevice: "1"
  # Storage layer must never be evicted by kubelet under node pressure
@@ -1,64 +0,0 @@
---
-# A SEPARATE CephFS whose only purpose is Woodpecker CI workspaces. Each
-# pipeline gets a PVC that is created, filled and deleted within ~2 minutes;
-# keeping that churn off `cephfs` spares its size-3 metadataPool.
-#
-# size 2, not 1: an unreplicated pool makes `ceph osd ok-to-stop` refuse for
-# every OSD, blocking all OSD maintenance cluster-wide. size 1 is viable only
-# once these pools have OSDs of their own.
-apiVersion: ceph.rook.io/v1
-kind: CephFilesystem
-metadata:
-  name: cephfs-ci
-  namespace: rook-ceph
-spec:
-  metadataPool:
-    replicated:
-      size: 2
-  dataPools:
-    - name: data0
-      replicated:
-        size: 2
-  metadataServer:
-    activeCount: 1
-    # No standby-replay, unlike `cephfs`. A standby exists there so a failed
-    # MDS does not hang every CephFS mount cluster-wide; here the only casualty
-    # is a running pipeline, and a second daemon to protect throwaway work is
-    # not worth its memory.
-    activeStandby: false
-    resources:
-      # Deliberately NOT the 4Gi that `cephfs` needs. That figure is driven by
-      # immich's millions of cached inodes; a CI workspace is a few thousand
-      # short-lived files that vanish with the pipeline.
-      requests:
-        cpu: 100m
-        memory: 512Mi
-      limits:
-        memory: 1Gi
---
-# CI-only class. `ceph-filesystem-ephemeral` is deliberately left pointing at
-# the main filesystem: velero/restore-test.sh uses it to restore REAL data for
-# verification, which must not land on unreplicated pools.
-#
-# A new class rather than repointing the old one — StorageClass.parameters are
-# immutable, so `fsName` cannot be edited in place.
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: ceph-filesystem-ci
-provisioner: rook-ceph.cephfs.csi.ceph.com
-parameters:
-  clusterID: rook-ceph
-  fsName: cephfs-ci
-  pool: cephfs-ci-data0
-  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
-  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
-  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
-  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
-  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
-  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
-# Delete, for the same reason ceph-filesystem-ephemeral uses it: with Retain,
-# every finished pipeline leaks a Released PV pinning a subvolume (191 of them
-# by 2026-07-16, issue #262 follow-up).
-reclaimPolicy: Delete
-allowVolumeExpansion: true
@@ -4,6 +4,5 @@ resources:
  - cephcluster.yaml
  - toolbox.yaml
  - storageclass.yaml
-  - cephfs-ci.yaml
  - volumesnapshotclass.yaml
  - objectstore.yaml
@@ -87,31 +87,15 @@ metadata:
  name: cephfs
  namespace: rook-ceph
 spec:
-  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
-  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
-  # three conversions rather than being toggled per node: each change
-  # re-replicates the pool, and the window reopens with the next node anyway.
-  #
-  # The standing trade for this pool is size 3: it holds ~253 MiB against the
-  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
-  # the entire reason data0 dropped to 2 — while losing it would destroy the
-  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
-  # quarter of a gigabyte to keep that intact is the right trade in steady
-  # state.
-  #
-  # It does not hold while one of three hosts is out. size 3 on two remaining
-  # hosts leaves every PG undersized at two copies with min_size 2, so a single
-  # further OSD loss drops the pool below min_size and blocks IO on every
-  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
-  # instead, and a further loss degrades rather than blocks — availability
-  # bought at the cost of accepting writes onto a single copy while degraded.
-  #
-  # See RUNBOOK-baremetal-conversion.md.
+  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
+  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
+  # measurable to drive wear — the entire reason for the size reduction — while
+  # losing it would destroy the whole filesystem, not just the objects on the
+  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
+  # trade; the wear saving comes from data0.
  metadataPool:
    replicated:
-      size: 2
-    parameters:
-      min_size: "1"
+      size: 3
  dataPools:
    - name: data0
      # size 2 / min_size 1 for the same reasons as replicapool above — see
@@ -164,9 +164,7 @@ spec:
            name: rook-config-override
            optional: true
        - name: ceph-config
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
@@ -1,30 +1,19 @@
 ---
-# Monthly Velero restore-test CronJob — 08:00 UTC on the 1st.
+# Weekly Velero restore-test CronJob.
 #
-# The sweep restores every discovered schedule, immich's ~180 GiB library
-# included. At `size: 2` that is ~360 GiB raw, ~120 GiB per OSD, against a
-# ~65 GB/day weekday baseline on three QLC NVMe at 81-87% endurance used —
-# ~47% of each drive's weekly write budget. Monthly keeps every restore path
-# exercised and cuts that share ~4x.
-#
-# 08:00 — *after* that day's own daily backups have finished (paperless
-# 03:15 through apps 07:15, each normally ~1 min). The previous 02:00 slot
-# was chosen as "well before the next day's backups", which missed that the
-# daily schedules fire on the same day: a sweep starting at 02:00 and
-# running ~5 h sat on top of the entire 03:15-07:15 backup window and
+# Sunday 08:00 UTC — *after* Sunday's own daily backups have finished
+# (paperless 03:15 through apps 07:15, each normally ~1 min). The previous
+# 02:00 slot was chosen as "well before Monday's backups", which missed
+# that the daily schedules also fire on Sunday: a sweep starting at 02:00
+# and running ~5 h sat on top of the entire 03:15-07:15 backup window and
 # starved it of node-agent DataUpload concurrency. Observed effect on
 # 2026-07-19 and 2026-07-26 — audiobookshelf-daily took 2 h 18 m and
 # 2 h 33 m respectively against a ~1 min baseline, with the books
-# DataUpload not starting until 06:18. Both are Sundays, the only day the
-# sweep ran then; no other day shows it.
+# DataUpload not starting until 06:18. Both are Sundays; no other day
+# shows it.
 #
-# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of the
-# next day's 03:15.
-#
-# Day-of-month rather than a fixed weekday: Kubernetes parses cron with
-# robfig/cron's standard parser, which has no `#` nth-weekday operator, and
-# restricting both day fields (`0 8 1-7 * 0`) ORs them — every Sunday plus
-# the 1st through 7th.
+# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of
+# Monday's 03:15.
 #
 # Concurrency policy Forbid: the immich restore (~180 GiB, millions of
 # small files) alone can run multiple hours, so the full sweep is sized
@@ -40,7 +29,7 @@ metadata:
  name: restore-test
  namespace: velero
 spec:
-  schedule: "0 8 1 * *"
+  schedule: "0 8 * * 0"
  timeZone: Etc/UTC
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
@@ -50,7 +39,7 @@ spec:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 21600  # 6 h hard cap (immich alone can take ~3 h)
-      ttlSecondsAfterFinished: 3456000  # 40 d — enough to inspect after the next monthly run
+      ttlSecondsAfterFinished: 1209600  # 14 d — enough to inspect after the next weekly run
      template:
        spec:
          serviceAccountName: restore-test
@@ -96,6 +85,4 @@ spec:
                name: restore-test-script
                defaultMode: 0755
            - name: tmp
-              emptyDir:
-                medium: Memory
-                sizeLimit: 16Mi
+              emptyDir: {}
@@ -1,5 +1,5 @@
 ---
-# RBAC for the monthly restore-test CronJob.
+# RBAC for the weekly restore-test CronJob.
 #
 # Permissions required by restore-test.sh:
 #   - velero.io: read schedules/backups/BSL, create+delete restores
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Monthly verification that Velero PVC backups are actually restorable.
+# Weekly verification that Velero PVC backups are actually restorable.
 #
 # What this exercises (per service):
 #   1. Velero Restore reaches phase=Completed against yesterday's
@@ -53,7 +53,7 @@ check() {
 # and ceph-filesystem StorageClasses ship with reclaimPolicy: Retain so that
 # production PVC deletions can be undone manually. The restore-test has the
 # opposite need: every PVC it creates is throwaway and MUST not survive the
-# iteration. Without this, each run leaks O(100 GiB) of CephFS
+# iteration. Without this, each weekly run leaks O(100 GiB) of CephFS
 # subvolumes and the cluster eventually goes OSD_FULL (incident 2026-05-17).
 #
 # Why this still exists after ceph-*-ephemeral landed. Three categories of
@@ -1,129 +1,62 @@
-# k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
-# still Proxmox VMs. Every machine-specific selector below differs from theirs
-# for that reason: there are no virtio NICs, no PCI-passthrough layer, and all
-# four physical NICs and all three NVMe drives are visible to Talos at once.
-#
-# Networking uses the v1.12 multi-document config rather than the deprecated
-# machine.network.interfaces. That is deliberate: the docs do not define what
-# happens when the same link is configured by both schemas, so this file keeps
-# every link in one schema instead of straddling them.
 machine:
  network:
    hostname: k8s-1
+    # Shared control-plane VIP — etcd-elected leader hosts 10.2.40.60 as a
+    # secondary IP. Exposed via UniFi DNS as k8s.renner.uno.
+    #
+    # Two virtio NICs (mgmt + v30iot). `driver: virtio_net` would match
+    # both, so each is pinned by its PCI bus path:
+    #   - 0000:06:12.0 — original mgmt NIC (the only virtio before)
+    #   - 0000:06:13.0 — extra_network_bridges = ["v30iot"] in tofu
+    # Bus paths are consistent across k8s-{1,2,3}; verified live 2026-05-13.
+    # The X710 selector stays driver-based (i40e is unambiguous) and
+    # tolerates kernel renaming.
+    interfaces:
+      - deviceSelector:
+          busPath: "0000:06:12.0"
+        dhcp: true
+        vip:
+          ip: 10.2.40.60
+      # v30iot: LINK ONLY — no host addressing. The NIC is the macvlan
+      # parent for the v30iot pod attachments (matter/otbr/shelly-manager),
+      # whose children do their own DHCP on the L2; host addresses were
+      # never needed for that. Host DHCP (formerly dhcp + routeMetric 4096)
+      # installed a connected 10.2.30.0/24 route, so node replies to
+      # IoT-subnet clients bypassed the UDM and broke every UDM-DNATed
+      # flow (hairpin NAT: un-translated SYN-ACK -> client kernel RST,
+      # verified live 2026-07-23). The purpose the old comment claimed —
+      # Cilium L2 announce for an IoT LB pool — does not exist (no
+      # L2AnnouncementPolicy, no such pool). v6 RA addressing on this NIC
+      # is disabled via machine.sysctls for the same reason.
+      - deviceSelector:
+          busPath: "0000:06:13.0"
+        dhcp: false
+      # Dedicated Ceph cluster network on the X710 SFP+ port (PCI passthrough,
+      # mapping x710-port1 in opentofu/vm_150_k8s_1.tf). Separate /29 on VLAN 49,
+      # MTU 9000, no default route — used by Rook for OSD↔OSD replication via
+      # CephCluster.spec.network.addressRanges.cluster.
+      - deviceSelector:
+          driver: i40e
+        addresses:
+          - 10.99.0.1/29
+        mtu: 9000
  sysctls:
-    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
-    # the macvlan parent for the v30iot pod attachments
-    # (matter/otbr/shelly-manager/music-assistant), whose children do their own
-    # DHCP on that L2. Host addresses were never needed and are actively
-    # harmful: a connected 10.2.30.0/24 route makes node replies to IoT-subnet
-    # clients bypass the UDM, which breaks every UDM-DNATed flow (hairpin NAT:
-    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
-    # This disables v6 entirely on that link, including link-local.
+    # v30iot NIC carries no host addressing (see interfaces above); this
+    # disables v6 entirely (incl. link-local) so the host is unreachable on that L2.
+    #
+    # Keyed on the STABLE predictable name `ens19` (busPath 0000:06:13.0 =
+    # v30iot). Predictable naming is re-enabled via the Image Factory
+    # schematic (net.ifnames=1; see .woodpecker/deploy-infrastructure.yml),
+    # so `ens19` is topology-derived and does NOT shuffle across reboots —
+    # unlike the former `eth0/1/2` (driver-probe order) which renamed on a
+    # k8s-3 reboot and silently mis-targeted this sysctl onto mgmt. The four
+    # v30iot macvlan NADs key on the same `ens19`.
    net.ipv6.conf.ens19.disable_ipv6: "1"
  install:
-    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
-    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
-    # Ceph OSD, and the retired Crucial P3 that carries EPHEMERAL (see the
-    # VolumeConfig below). Every one of them is selected by serial. Nothing in
-    # this file may use a broad selector: `disk.transport == 'nvme'` matches all
-    # three, and the failure mode is Talos taking the OSD drive.
-    diskSelector:
-      serial: 50026B7382B48B86
+    disk: /dev/vda
  certSANs:
    - k8s-1.srv.renner.uno
  nodeLabels:
    topology.kubernetes.io/zone: jupiter
 cluster:
  allowSchedulingOnControlPlanes: true
---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk.
-#
-# That drive is at 87% of its rated endurance and was this node's Ceph OSD until
-# the PM983 replaced it. /var is exactly the right role for a disk in that
-# state: containerd's image and snapshot store, pod logs and the Ceph mon store
-# are all reconstructible, so the drive is consumable here in a way it is not as
-# an OSD. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had,
-# which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
-# they absorb another node's pods.
-#
-# THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
-# solely when the volume does not yet exist; a config applied after EPHEMERAL
-# has been created elsewhere is silently ignored, and relocating it then
-# requires wiping the disk and booting install media again.
-#
-# Matched by serial deliberately. `disk.transport == 'nvme' && !system_disk`
-# would match the PM983 as well, and losing that coin flip means Talos formats
-# the OSD drive.
-apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '2332E86B749A'
-  minSize: 50GiB
-  grow: true
---
-# X710 port 0 carries the switch trunk. It takes no addressing itself — the
-# port's untagged VLAN is the 192.168.8.0/24 network, which this node has no
-# business being on — so management and v30iot both ride it as tagged VLANs.
-#
-# Links are addressed by predictable kernel name. Those derive from PCI
-# topology, not probe order — though not immune: adding the third NVMe
-# renumbered the buses and moved the unused second I226 from enp89s0 to
-# enp90s0. Neither name used here was affected.
-#
-# LinkAliasConfig would be the better way to pin these, and is worth
-# reintroducing — but only once it is proven. Aliasing both X710 ports by
-# permanent MAC and using the aliases as VLAN parents produced a node with no
-# address on any interface: the aliases were never created, so every document
-# referencing them did nothing, including the static address on port 1. Verify
-# any alias-based revision with `talosctl apply-config --mode=try`, which rolls
-# back on its own, before making it permanent.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f0np0
-up: true
---
-# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network. The k8s-1 VM reached 10.2.40.174 over this same physical port via
-# Proxmox's v40srv VNet (vmbr0.40), so the switch trunks VLAN 40 here.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: mgmt0
-vlanID: 40
-parent: enp2s0f0np0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: mgmt0
---
-# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
-# k8s.renner.uno.
-apiVersion: v1alpha1
-kind: Layer2VIPConfig
-name: 10.2.40.60
-link: mgmt0
---
-# Named ens19 on purpose. The four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
-# that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
-# serve both VM and bare-metal nodes for as long as the cluster is mixed.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ens19
-vlanID: 30
-parent: enp2s0f0np0
-up: true
---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f1np1
-up: true
-mtu: 9000
-addresses:
-  - address: 10.99.0.1/29
@@ -1,99 +1,31 @@
-# k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
-# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
-# VM-era layout until neptune is converted.
 machine:
  network:
    hostname: k8s-2
+    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
+    interfaces:
+      - deviceSelector:
+          busPath: "0000:06:12.0"
+        dhcp: true
+        vip:
+          ip: 10.2.40.60
+      # v30iot: link only, no host addressing — see k8s-1.yaml.
+      - deviceSelector:
+          busPath: "0000:06:13.0"
+        dhcp: false
+      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
+      - deviceSelector:
+          driver: i40e
+        addresses:
+          - 10.99.0.2/29
+        mtu: 9000
  sysctls:
-    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It is the
-    # macvlan parent for the v30iot pod attachments; host addresses on that L2
-    # make node replies bypass the UDM and break every UDM-DNATed flow.
+    # v30iot NIC carries no host addressing — see k8s-1.yaml.
    net.ipv6.conf.ens19.disable_ipv6: "1"
  install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
-    diskSelector:
-      serial: 50026B7382DA5F6E
+    disk: /dev/vda
  certSANs:
    - k8s-2.srv.renner.uno
  nodeLabels:
    topology.kubernetes.io/zone: saturn
 cluster:
  allowSchedulingOnControlPlanes: true
---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 81% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had.
-#
-# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
-# solely when the volume does not yet exist; applied later it is silently
-# ignored and relocating needs a disk wipe and a boot from install media.
-#
-# Matched by serial deliberately: a broad selector such as
-# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
-apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '2332E86B74BC'
-  minSize: 50GiB
-  grow: true
---
-# X710 port 0 carries the switch trunk and takes no addressing itself — the
-# port's untagged VLAN is not one this node belongs on. Management and v30iot
-# both ride it as tagged VLANs.
-#
-# Addressed by predictable kernel name, not by LinkAliasConfig. Aliases are
-# valid VLAN parents, but a selector matching zero links is skipped silently
-# and Talos disables default DHCP machine-wide as soon as any new-style network
-# document exists — so a silent miss leaves the node with no address on any
-# interface. See RUNBOOK-baremetal-conversion.md.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f0np0
-up: true
---
-# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: mgmt0
-vlanID: 40
-parent: enp2s0f0np0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: mgmt0
---
-# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
-# k8s.renner.uno.
-apiVersion: v1alpha1
-kind: Layer2VIPConfig
-name: 10.2.40.60
-link: mgmt0
---
-# Named ens19 on purpose: the four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-3 still presents the VLAN as a virtio NIC of that
-# name. VLANConfig.name is free-form, so one set of NADs serves both.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ens19
-vlanID: 30
-parent: enp2s0f0np0
-up: true
---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. OSD<->OSD replication only. Not routed off its own
-# segment and deliberately has no default route.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f1np1
-up: true
-mtu: 9000
-addresses:
-  - address: 10.99.0.2/29
@@ -0,0 +1,51 @@
+module "k8s_1" {
+  source = "./modules/proxmox_vm"
+
+  name        = "k8s-1.srv.renner.uno"
+  target_node = "jupiter"
+  vmid        = 150
+  memory      = 49152
+  cores       = 8
+
+  template_name = null
+  cloud_init    = false
+  full_clone    = false
+  bios          = "ovmf"
+  machine       = "q35"
+  boot          = "order=virtio0"
+  agent_enabled = true
+
+  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
+  # VLAN from this node (previously only k8s-edge had v30iot, so any
+  # IoT-pool LB lease landing here was unreachable from devices).
+  extra_network_bridges = ["v30iot"]
+
+  disks = [
+    {
+      storage = "local-zfs"
+      size    = "64G"
+      format  = "raw"
+      slot    = "virtio0"
+      file_id = local.talos_image
+    }
+  ]
+
+  hostpci = [
+    {
+      mapping = "osd-ssd"
+    },
+    {
+      mapping = "x710-port1"
+    }
+  ]
+
+  tags            = "kubernetes;talos;controlplane"
+  ssh_public_keys = local.common_ssh_keys
+
+  vm_state = "running"
+
+  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
+  # non-responsive Talos guest can't hang the destroy for 30 minutes.
+  stop_on_destroy = true
+}
+
@@ -0,0 +1,50 @@
+module "k8s_2" {
+  source = "./modules/proxmox_vm"
+
+  name        = "k8s-2.srv.renner.uno"
+  target_node = "saturn"
+  vmid        = 151
+  memory      = 49152
+  cores       = 8
+
+  template_name = null
+  cloud_init    = false
+  full_clone    = false
+  bios          = "ovmf"
+  machine       = "q35"
+  boot          = "order=virtio0"
+  agent_enabled = true
+
+  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
+  # VLAN from this node (previously only k8s-edge had v30iot, so any
+  # IoT-pool LB lease landing here was unreachable from devices).
+  extra_network_bridges = ["v30iot"]
+
+  disks = [
+    {
+      storage = "local-zfs"
+      size    = "64G"
+      format  = "raw"
+      slot    = "virtio0"
+      file_id = local.talos_image
+    }
+  ]
+
+  hostpci = [
+    {
+      mapping = "osd-ssd"
+    },
+    {
+      mapping = "x710-port1"
+    }
+  ]
+
+  tags            = "kubernetes;talos;controlplane"
+  ssh_public_keys = local.common_ssh_keys
+
+  vm_state = "running"
+
+  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
+  # non-responsive Talos guest can't hang the destroy for 30 minutes.
+  stop_on_destroy = true
+}