fix(hcloud): test nodes take the resolver list from site-homelab.yaml

site-homelab.yaml now carries a ResolverConfig (k8s-gateway, then the UDM). The hcloud-N patches are applied after it and each repeated a ResolverConfig with the UDM alone, which would drop k8s-gateway on the test nodes. Removed; the test nodes now resolve like production. hcloud_gw_mgmt_hostroutes read the UDM address from that removed document. It now reads it from site-homelab.yaml, skipping the k8s-gateway ClusterIP taken from its HelmRelease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(velero): restore Gitea first, the rest after it
2026-10-16 08:11:18 +02:00 · 2026-10-16 08:10:30 +02:00 · 2026-10-16 08:10:30 +02:00 · 2026-10-16 08:10:30 +02:00 · 2026-10-16 08:10:30 +02:00 · 2026-10-16 08:10:30 +02:00
32 changed files with 1524 additions and 221 deletions
@@ -10,10 +10,8 @@
 # - Name, namespace, accessModes, storageClassName and volumeMode must match
 #   the live claim exactly: they are immutable, a mismatch fails the whole
 #   gitea Kustomization (no change to the live claim).
-# - volumeName must stay: kustomize-controller has owned spec.volumeName
-#   since the namespace move pre-created this claim (eb23d3ac, #878).
-#   Without it, server-side apply tries to clear the field and the API server
-#   rejects the whole claim as an immutable-spec change.
+# - No volumeName: the gitea Kustomization ignores /spec/volumeName on this
+#   claim, so the bound volume is kept and a fresh cluster binds a new one.
 # - Grow only. A smaller value is rejected by the API server.
 # - prune: disabled, so removing this file from git never deletes the claim
 #   and its data. To hand ownership back, delete the file; the claim stays.
@@ -33,7 +31,6 @@ spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ceph-block
  volumeMode: Filesystem
-  volumeName: pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be
  resources:
    requests:
      storage: 60Gi
@@ -131,6 +131,7 @@ on_exit() {
    # behind are its product. `hcloudclean` is what deletes them.
    # Only when this run actually booked machines. The function is always
    # defined, so testing for it fired on the qemu phase too.
+    [[ -n "${HCLOUD_GW_ADDR:-}" ]] && hcloud_gw_ssh_close
    if [[ "${HCLOUD_PROVISIONED:-0}" == "1" ]]; then
        echo -e "${YELLOW}Hetzner machines are UP and billed until deleted.${NC}"
        echo "  ./kubernetes/bootstrap.sh hcloudclean   # when finished"
@@ -217,7 +218,7 @@ OPENBAO_SECRETS_DIR="${OPENBAO_SECRETS_DIR:-}"
 # that cannot be told apart from a fault. Its non-zero exit propagates, which is
 # what makes require_secrets print the reason instead.
 git_token_for_source() {
-    if [[ "$FLUX_SOURCE" == "github" ]]; then
+    if [[ "$FLUX_SOURCE" == "github" || "$FLUX_SOURCE" == "restore-test" ]]; then
        if [[ -n "${GITHUB_TOKEN:-}" ]]; then
            printf '%s' "$GITHUB_TOKEN"
        else
@@ -255,13 +256,13 @@ OPENBAO_REQUIRED_SECRETS=(
    # rest, because a rebuild that discovers it missing has already wiped the
    # disks.
    "ghcr-pull|GHCR pull token for the image mirror|pass show homelab/github/k8s-package-pull"
-    # Only the hcloudtest and hcloudclean phases ask for this one, by name, so a
-    # rebuild of the real cluster never demands it.
-    #
-    # Note the path: homelab/hetzner/, spelled correctly, while the two restore
-    # keys above live under homelab/hetztner/. Two folders, one typo, both real
-    # — do not "fix" one of these paths without moving the entry first.
-    "hcloud-token|Hetzner Cloud API token for the throwaway test cluster|pass show homelab/hetzner/cloud-api-secret"
+)
+# Asked for by name only, never by a bare require_secrets: a rebuild of the
+# real cluster cannot depend on the OpenBao it is rebuilding. Scoped to the
+# Hetzner project homelab-backup-verify, which holds nothing but throwaway
+# machines.
+HCLOUD_REQUIRED_SECRETS=(
+    "hcloud-token|Hetzner Cloud API token for the throwaway test cluster|bao kv get -mount=homelab -field=token infrastructure/hcloud-backup-verify"
 )

 # Collects all of them up front, into one directory, so everything after this
@@ -288,7 +289,7 @@ require_secrets() {
    local entry name what how
    local -a wanted=()
    if [[ $# -gt 0 ]]; then
-        for entry in "${OPENBAO_REQUIRED_SECRETS[@]}"; do
+        for entry in "${OPENBAO_REQUIRED_SECRETS[@]}" "${HCLOUD_REQUIRED_SECRETS[@]}"; do
            for name in "$@"; do
                [[ "${entry%%|*}" == "$name" ]] && wanted+=("$entry")
            done
@@ -407,13 +408,14 @@ read_secret() {
 #
 # WHAT IT DOES NOT COVER.
 #
-# Everything downstream of Multus. The provider is not the limit — a cloud
-# server takes up to 16 volumes and 3 private networks — but ceph-san binds to
-# the kernel link name enp2s0f1np1, and that name cannot be manufactured here
-# without assuming a Hetzner private network passes VLAN tags. See
-# talos/patches/site-hcloud.yaml. Nor Flux and everything after it: that phase
-# needs the OpenBao seal key, which deliberately never reaches CI, and this
-# test is meant to run unattended.
+# Flux and everything after it: that phase needs the OpenBao seal key, which
+# deliberately never reaches CI.
+#
+# The network is the production one rebuilt in WireGuard: Hetzner networks
+# are layer 3 and IPv4-only, so each production segment is a WireGuard mesh
+# over a private underlay, with the production link names and addresses, and
+# hcloud-gw stands in for the UDM. The nodes take site-homelab.yaml unchanged.
+# See hcloud-site.sh.
 # Talos version and Image Factory schematic, read out of the installer image in
 # cluster.yaml. Both throwaway-cluster phases need them and neither may state
 # them: a second copy is a copy that goes stale, and the day it did the test
@@ -444,22 +446,19 @@ hcloud_volumes_for_node() {
        gib=${size%GiB}
        [[ "$gib" =~ ^[0-9]+$ ]] || fail "$patch: cannot read volume size '$size' — state it as <n>GiB"
        echo $(( (gib * 1074 + 999) / 1000 ))
-    done < <(yq -r 'select(.kind == "RawVolumeConfig") | .provisioning.minSize' "$patch")
+    done < <(yq -r 'select(.kind == "RawVolumeConfig" and .provisioning.diskSelector.match != "system_disk") | .provisioning.minSize' "$patch")
 }

 # Deletes everything this phase created, and NOTHING else.
 #
-# That distinction is not theoretical: the project this runs in also holds
-# netbird.renner.uno, which has been up for months and is the VPN way in. The
-# label is the only thing separating a throwaway server from it — and
-# `hcloud server list -l ""` lists EVERY server, so an unset HCLOUD_LABEL would
-# turn this function into a project wipe. Hence the guard: the label is checked
-# before it is used, and every candidate must also carry the name prefix this
-# phase assigns. Two independent conditions, because one of them is a variable.
-#
-# A separate Hetzner project, with a token scoped to it, would make this
-# structural rather than defensive. Worth doing before this runs unattended.
+# The project (homelab-backup-verify) also runs the backup-verify CronJob's
+# machines. `hcloud server list -l ""` lists EVERY server, so an unset
+# HCLOUD_LABEL would turn this function into a project wipe: the label is
+# checked before it is used, and every candidate must also carry the name
+# prefix this phase assigns.
 HCLOUD_NAME_PREFIX="hcloud"
+# shellcheck source=hcloud-site.sh
+source "$REPO_ROOT/kubernetes/hcloud-site.sh"

 cleanup_hcloudtest() {
    command -v hcloud >/dev/null || return 0
@@ -490,6 +489,11 @@ cleanup_hcloudtest() {
        hcloud volume delete "$id" >/dev/null \
            || echo "  WARNING: volume $id not deleted — check the console" >&2
    done
+    for id in $(hcloud ssh-key list -l "$HCLOUD_LABEL" -o noheader -o columns=id); do
+        echo "  deleting ssh key $id"
+        hcloud ssh-key delete "$id" >/dev/null \
+            || echo "  WARNING: ssh key $id not deleted — check the console" >&2
+    done
    # Last: a network with a server still attached cannot be deleted, and a
    # leftover one takes its name with it, so the next run cannot recreate it.
    for id in $(hcloud network list -l "$HCLOUD_LABEL" -o noheader -o columns=id); do
@@ -499,6 +503,17 @@ cleanup_hcloudtest() {
    done
 }

+# The restore test's source, as restore-test/flux-root.yaml names it: sets
+# restore_url and restore_repo (<owner>/<repo>).
+restore_test_repo() {
+    command -v yq >/dev/null || fail "yq not found (reads restore-test/flux-root.yaml)"
+    restore_url=$(yq -r 'select(.kind == "GitRepository" and .metadata.name == "restore-test") | .spec.url' \
+        "$REPO_ROOT/kubernetes/restore-test/flux-root.yaml")
+    [[ "$restore_url" =~ ^ssh://git@github\.com/([^/]+)/([^/]+)\.git$ ]] \
+        || fail "restore-test/flux-root.yaml names '$restore_url', not ssh://git@github.com/<owner>/<repo>.git"
+    restore_repo="${BASH_REMATCH[1]}/${BASH_REMATCH[2]}"
+}
+
 # Sweep only. For the case where a previous run was killed hard enough that its
 # own trap never ran, and for a cron job to call before it starts.
 if [[ "$PHASE" == "hcloudclean" ]]; then
@@ -507,6 +522,16 @@ if [[ "$PHASE" == "hcloudclean" ]]; then
    require_secrets hcloud-token
    HCLOUD_TOKEN=$(read_secret hcloud-token); export HCLOUD_TOKEN
    cleanup_hcloudtest
+    # The restore test's read-only deploy keys on the GitHub mirror.
+    if command -v gh >/dev/null; then
+        restore_test_repo
+        for key_id in $(gh api "repos/$restore_repo/keys" \
+                --jq '.[] | select(.title | startswith("restore-test ")) | .id'); do
+            echo "  deleting deploy key $key_id"
+            gh api -X DELETE "repos/$restore_repo/keys/$key_id" \
+                || echo "  WARNING: deploy key $key_id not deleted — check the repo settings" >&2
+        done
+    fi
    ok
    exit 0
 fi
@@ -518,6 +543,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
    command -v yq >/dev/null       || fail "yq not found"
    command -v jq >/dev/null       || fail "jq not found (needed to read the SAN pool and server attachments)"
    command -v curl >/dev/null     || fail "curl not found (needed to learn this host's public address)"
+    command -v wg >/dev/null       || fail "wg not found (wireguard-tools, needed to generate the mesh keys)"
+    command -v bao >/dev/null      || fail "bao not found (the Hetzner token is read from OpenBao)"

    # Three control planes, all carrying workloads — the production shape. One
    # machine is a node, not a cluster: etcd has no quorum to lose and nothing
@@ -527,7 +554,6 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
    for n in "${HCLOUD_NODES[@]}"; do
        [[ -f "$TALOS_DIR/patches/$n.yaml" ]] || fail "$TALOS_DIR/patches/$n.yaml not found"
    done
-    [[ -f "$TALOS_DIR/patches/site-hcloud.yaml" ]] || fail "site-hcloud.yaml not found"

    require_secrets hcloud-token ghcr-pull
    HCLOUD_TOKEN=$(read_secret hcloud-token); export HCLOUD_TOKEN
@@ -536,18 +562,11 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
    # this one, adopted below. The names are fixed, so duplicates cannot
    # accumulate. `hcloudclean` is what deletes.

-    # The Talos snapshot is a prerequisite, not something this builds: making it
-    # means booting a server into rescue and writing the raw image, which is a
-    # one-off and needs a different set of permissions. Fail with the recipe
-    # rather than silently booting Debian and timing out on the Talos API.
    step 0 "Looking up the Talos snapshot..."
    talos_image_coords
-    HCLOUD_IMAGE=$(hcloud image list --type snapshot -l os=talos -o noheader -o columns=id | head -1)
-    [[ -n "$HCLOUD_IMAGE" ]] || fail "no snapshot labelled os=talos. Create one once — boot any server into rescue, then:
-    wget -O /tmp/talos.raw.xz https://factory.talos.dev/image/$VM_SCHEMATIC/$VM_VERSION/hcloud-amd64.raw.xz
-    xz -d -c /tmp/talos.raw.xz | dd of=/dev/sda && sync && shutdown -h now
-  then snapshot that server in the console and label the snapshot os=talos.
-  The schematic and version above are this cluster's, read from cluster.yaml."
+    HCLOUD_IMAGE=""
+    hcloud_talos_snapshot
+    [[ -n "$HCLOUD_IMAGE" ]] || fail "no Talos snapshot"
    echo "  snapshot $HCLOUD_IMAGE"

    # The Talos API is unauthenticated in maintenance mode, which is how
@@ -589,53 +608,43 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
    hcloud firewall add-rule "hcloudtest-$$" --direction in --protocol tcp --port 6443 \
        --source-ips "$fw_src" --description "Kubernetes API, this host only" >/dev/null \
        || fail "could not add the Kubernetes API rule"
+    hcloud firewall add-rule "hcloudtest-$$" --direction in --protocol tcp --port 22 \
+        --source-ips "$fw_src" --description "ssh to hcloud-gw, this host only" >/dev/null \
+        || fail "could not add the ssh rule"
    echo "  open to $fw_src only"
+    # Servers held from an earlier run get this firewall first; only then are
+    # the earlier runs' inbound firewalls (stale addresses, and a server takes
+    # only a few) detached and deleted, so no server is ever without one. Their
+    # egress firewalls stay until hcloud_apply_egress has attached this run's.
+    while read -r srv; do
+        [[ -n "$srv" ]] || continue
+        hcloud firewall apply-to-resource "hcloudtest-$$" --type server --server "$srv" >/dev/null \
+            || fail "could not attach this run's firewall to $srv"
+    done < <(hcloud server list -l "$HCLOUD_LABEL" -o noheader -o columns=name)
+    while read -r fw_id fw_name; do
+        [[ -n "$fw_id" && "$fw_name" != "hcloudtest-$$" && "$fw_name" != hcloudtest-*-egress-* ]] || continue
+        while read -r srv_id; do
+            [[ -n "$srv_id" ]] || continue
+            hcloud firewall remove-from-resource "$fw_id" --type server --server "$srv_id" >/dev/null \
+                || fail "could not detach $fw_name from server $srv_id"
+        done < <(hcloud firewall describe "$fw_id" -o json | jq -r '.applied_to[] | select(.type == "server") | .server.id')
+        hcloud firewall delete "$fw_id" >/dev/null || fail "could not delete the previous firewall $fw_name"
+        echo "  removed $fw_name"
+    done < <(hcloud firewall list -l "$HCLOUD_LABEL" -o noheader -o columns=id,name)
    ok

-    # cx43 — 16 GB, 8 vCPU. Sized from what the workload needs, not from the
-    # Talos guide, which names cx22 as a minimal example (and that type no
-    # longer exists; `server create` rejects it outright).
+    # Default type cpx52 (12 vCPU, 24 GB, 480 GB local disk): the hcloud-N
+    # patches put a 345GiB OSD partition beside a 60GiB EPHEMERAL on the
+    # system disk, and three of them plus the cx23 gateway fit the project's
+    # 40-core quota. Production's pods request ~71 GiB without the GPU apps.
    #
-    # cx23 was tried first, on the reasoning that it is cx22's successor. It has
-    # 4 GB, 3.2 GB of it allocatable, and the three nodes sat at 59-78% of that
-    # with the infrastructure skeleton alone — no Ceph OSDs, no OpenBao running,
-    # not one app. Limits were already at 134-207% of allocatable and both
-    # kube-controller-manager and kube-scheduler had restarted twice.
-    #
-    # The local rehearsal is the reference: it runs a 16GiB control plane and
-    # three 8GiB workers, 40GiB in total, and needed every step of the way up
-    # from 6GiB to get there. Three nodes here carry the same workload plus a
-    # control plane each, so 48GiB total is the comparable figure.
-    #
-    # Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which
-    # would not match the amd64 schematic this cluster installs. That leaves the
-    # cx*/cpx*2 families in the German locations.
-    # Two segments only, unlike the homelab's split in
-    # NETWORK-traffic-classes.md: here the node address carries management,
-    # cluster-internal and external alike, and Ceph replication has its own.
-    #
-    # The order is load-bearing: it decides which kernel interface each segment
-    # becomes, and the machine config names those interfaces.
-    HCLOUD_SEGMENTS=(cluster storage)
-    # Segment ranges are read from what the cluster declares, never invented
-    # here — the disks' rule applied to networks:
-    #   cluster  the subnet the site patch pins node addresses to.
-    #   storage  the SAN pool the ceph-san attachment allocates from, widened
-    #            to the containing /24 because the fabric wants a subnet.
-    # The fabric reserves the first address of a network range as its gateway.
-    # For the storage net that is the SAN pool's first address — a collision
-    # that matters only if cross-node SAN traffic on this site is ever
-    # attempted, and is left unresolved until it is.
+    # underlay  carries the WireGuard meshes between the machines (hcloud-site.sh),
+    #           as enp7s0. There is no storage network: the Ceph SAN cannot exist
+    #           on Hetzner (restore-test/flux-root.yaml).
+    HCLOUD_SEGMENTS=(underlay)
    seg_range() {
        case "$1" in
-            cluster)
-                yq -r '.machine.kubelet.nodeIP.validSubnets[0]' "$TALOS_DIR/patches/site-hcloud.yaml" ;;
-            storage)
-                local pool
-                pool=$(yq -r '.spec.config' "$REPO_ROOT/kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml" | jq -r '.ipam.range')
-                [[ "$pool" == *.0/2[4-9] ]] \
-                    || fail "cannot derive a /24 from the SAN pool '$pool'"
-                printf '%s/24' "${pool%.*}.0" ;;
+            underlay) printf '%s' "$HCLOUD_UNDERLAY_RANGE" ;;
        esac
    }
    # Read from the location rather than stated a second time.
@@ -671,8 +680,46 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
    done
    ok

+    step 0 "Generating the WireGuard meshes..."
+    hcloud_wg_keys
+    HCLOUD_WG_DIR="$OPENBAO_SECRETS_DIR/wg"
+    mkdir -p "$HCLOUD_WG_DIR"
+    for n in "${HCLOUD_NODES[@]}"; do
+        hcloud_node_wg_patch "$n"
+    done
+    ok
+
    # From here on machines exist and are billed; on_exit says so.
    HCLOUD_PROVISIONED=1
+    step 0 "Creating $HCLOUD_GW_NAME (UDM stand-in)..."
+    if hcloud server list -l "$HCLOUD_LABEL" -o noheader -o columns=name | grep -qx "$HCLOUD_GW_NAME"; then
+        echo "  replacing the previous $HCLOUD_GW_NAME: its keys belong to the previous run"
+        hcloud server delete "$HCLOUD_GW_NAME" >/dev/null || fail "could not delete the previous $HCLOUD_GW_NAME"
+    fi
+    if hcloud ssh-key list -l "$HCLOUD_LABEL" -o noheader -o columns=name | grep -qx "$HCLOUD_GW_NAME"; then
+        hcloud ssh-key delete "$HCLOUD_GW_NAME" >/dev/null || fail "could not replace the $HCLOUD_GW_NAME ssh key"
+    fi
+    mkdir -p "$HCLOUD_STATE_DIR"
+    chmod 700 "$HCLOUD_STATE_DIR"
+    rm -f "$HCLOUD_STATE_DIR/gw-ssh" "$HCLOUD_STATE_DIR/gw-ssh.pub" "$HCLOUD_STATE_DIR/gw-known-hosts"
+    ssh-keygen -q -t ed25519 -N '' -C "$HCLOUD_GW_NAME" -f "$HCLOUD_STATE_DIR/gw-ssh"
+    hcloud ssh-key create --name "$HCLOUD_GW_NAME" --label "$HCLOUD_LABEL" \
+        --public-key-from-file "$HCLOUD_STATE_DIR/gw-ssh.pub" >/dev/null \
+        || fail "could not upload the $HCLOUD_GW_NAME ssh key"
+    hcloud_gw_userdata
+    gw_userdata="$OPENBAO_SECRETS_DIR/gw-userdata.yaml"
+    hcloud server create --name "$HCLOUD_GW_NAME" --type "${HCLOUD_GW_TYPE:-cx23}" --image debian-12 \
+        --location "${HCLOUD_LOCATION:-fsn1}" --ssh-key "$HCLOUD_GW_NAME" --firewall "hcloudtest-$$" \
+        --label "$HCLOUD_LABEL" --user-data-from-file "$gw_userdata" --start-after-create=false >/dev/null \
+        || fail "could not create $HCLOUD_GW_NAME"
+    # Attached before first boot, so cloud-init configures the NIC from metadata.
+    hcloud server attach-to-network "$HCLOUD_GW_NAME" --network hcloudtest-underlay \
+        --ip "$HCLOUD_GW_UNDERLAY_IP" >/dev/null || fail "could not attach $HCLOUD_GW_NAME to the underlay"
+    hcloud server poweron "$HCLOUD_GW_NAME" >/dev/null || fail "could not start $HCLOUD_GW_NAME"
+    HCLOUD_GW_ADDR=$(hcloud server ip "$HCLOUD_GW_NAME") || fail "could not read the address of $HCLOUD_GW_NAME"
+    echo "  $HCLOUD_GW_NAME at $HCLOUD_GW_ADDR, underlay $HCLOUD_GW_UNDERLAY_IP"
+    ok
+
    step 0 "Creating ${#HCLOUD_NODES[@]} server(s)..."
    NODE_ADDRS=()
    # Servers this test already holds. Capacity is the binding constraint, so a
@@ -691,12 +738,7 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
            echo "  $n: reusing the server we already hold"
            hcloud server rebuild "$n" --image "$HCLOUD_IMAGE" >/dev/null \
                || fail "could not rebuild $n back into maintenance mode"
-            # The firewall is named per run (hcloudtest-$$), so an adopted
-            # server is still attached to the previous run's. Attach this run's
-            # too — without it the Talos API is unreachable once the old one is
-            # cleaned up, and the addresses in it are last run's anyway.
-            hcloud firewall apply-to-resource "hcloudtest-$$" --type server --server "$n" >/dev/null \
-                || fail "could not attach this run's firewall to $n"
+            # This run's firewall was attached above, before the old ones went.
        else
            # Retried, because "error during placement (resource_unavailable)"
            # is the provider being momentarily out of that type in that
@@ -708,13 +750,13 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
            create_err=""
            for attempt in $(seq 1 30); do
                create_err=$(hcloud server create --name "$n" --image "$HCLOUD_IMAGE" \
-                    --type "${HCLOUD_TYPE:-cx43}" --location "${HCLOUD_LOCATION:-fsn1}" \
+                    --type "${HCLOUD_TYPE:-cpx52}" --location "${HCLOUD_LOCATION:-fsn1}" \
                    --firewall "hcloudtest-$$" --label "$HCLOUD_LABEL" 2>&1) && break
                grep -qi 'resource_unavailable\|placement' <<<"$create_err" \
                    || fail "could not create $n: ${create_err//$'\n'/ | }"
                [[ $attempt -eq 1 ]] && echo "  $n: no capacity yet, retrying"
                sleep 20
-                [[ $attempt -eq 30 ]] && fail "no capacity for ${HCLOUD_TYPE:-cx43} in ${HCLOUD_LOCATION:-fsn1} after 10 minutes"
+                [[ $attempt -eq 30 ]] && fail "no capacity for ${HCLOUD_TYPE:-cpx52} in ${HCLOUD_LOCATION:-fsn1} after 10 minutes"
            done
        fi
        addr=$(hcloud server ip "$n") || fail "could not read the address of $n"
@@ -730,9 +772,17 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
                | jq -e --argjson id "$net_id" 'any(.private_net[]; .network == $id)' >/dev/null; then
                continue
            fi
-            hcloud server attach-to-network "$n" --network "$net" >/dev/null \
+            attach_ip=()
+            [[ "$seg" == underlay ]] && attach_ip=(--ip "$(hcloud_underlay_ip "$n")")
+            hcloud server attach-to-network "$n" --network "$net" "${attach_ip[@]}" >/dev/null \
                || fail "could not attach $n to $net"
        done
+        # The meshes' endpoints are the fixed underlay addresses, and the node
+        # patches name the underlay NIC enp7s0: an adopted server with other
+        # attachments (an older run's networks) would never form them.
+        attached=$(hcloud server describe "$n" -o json | jq -r '[.private_net[] | .ip] | join(" ")')
+        [[ "$attached" == "$(hcloud_underlay_ip "$n")" ]] \
+            || fail "$n has private addresses '$attached', expected only $(hcloud_underlay_ip "$n") — run hcloudclean and start over"

        # The OSD device. Every node carries one, which is the production shape
        # — k8s-1/2/3 all hold storage — and is only possible here because
@@ -757,44 +807,46 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
            hcloud volume delete "$n-osd" >/dev/null \
                || fail "could not delete the previous OSD volume for $n"
        fi
+        yq -e 'select(.kind == "RawVolumeConfig" and .name == "osd-data")' "$TALOS_DIR/patches/$n.yaml" >/dev/null \
+            || fail "$n declares no osd-data RawVolumeConfig — a node with no OSD device is not this cluster's shape"
+        # A raw volume on the system disk needs no Hetzner volume.
        vol_size=$(hcloud_volumes_for_node "$n")
-        [[ -n "$vol_size" ]] || fail "$n declares no RawVolumeConfig — a node with no OSD device is not this cluster's shape"
-        hcloud volume create --name "$n-osd" --size "$vol_size" --server "$n" \
-            --label "$HCLOUD_LABEL" >/dev/null \
-            || fail "could not create the OSD volume for $n"
-
-        echo "  $n at $addr, with a ${vol_size}GB OSD volume"
+        if [[ -n "$vol_size" ]]; then
+            hcloud volume create --name "$n-osd" --size "$vol_size" --server "$n" \
+                --label "$HCLOUD_LABEL" >/dev/null \
+                || fail "could not create the OSD volume for $n"
+            echo "  $n at $addr, with a ${vol_size}GB OSD volume"
+        else
+            echo "  $n at $addr, OSD on the system disk"
+        fi
        NODE_ADDRS+=("$addr")
+        HCLOUD_PUBLIC_IP[$n]=$addr
+        hcloud_node_wg_patch "$n"
    done
    ok

-    # The nodes have to reach each other, and until now the firewall admitted
-    # only this host. One node did not care; three do — etcd peers on 2380, the
-    # apiserver, kubelet on 10250 and Cilium's VXLAN all run between them, and
-    # with the rules above alone etcd simply never forms a quorum.
-    #
-    # Added after creation because the addresses do not exist before it. Rules
-    # apply to an attached firewall immediately, so the ordering costs nothing.
-    #
-    # All ports between the nodes rather than an enumerated list: this is a
-    # throwaway cluster whose members are each other's only peers, and guessing
-    # the full port set of Talos plus Cilium is how a rebuild fails on the one
-    # port nobody wrote down. The outside still sees only 50000 and 6443, and
-    # only from this host.
-    step 0 "Opening the cluster to itself..."
-    node_src=$(IFS=,; printf '%s' "${NODE_ADDRS[*]/%//32}")
-    for proto in tcp udp; do
-        hcloud firewall add-rule "hcloudtest-$$" --direction in --protocol "$proto" \
-            --port any --source-ips "$node_src" \
-            --description "intra-cluster $proto" >/dev/null \
-            || fail "could not open $proto between the nodes"
+    # Node-to-node traffic runs over the underlay, which Hetzner firewalls do
+    # not filter; the public side admits only this host.
+    step 0 "Waiting for $HCLOUD_GW_NAME to finish cloud-init..."
+    # The UDM's IPS blocks a source that opens bursts of ssh connections
+    # (wiki: pve-ssh-port22-filtered-udm-ips-block). Give the gateway time to
+    # boot, then one connection, held open by ControlMaster; the cloud-init
+    # wait runs on the gateway.
+    sleep 90
+    for i in $(seq 1 4); do
+        hcloud_gw_ssh 'timeout 600 cloud-init status --wait >/dev/null; test -e /run/hcloud-gw-ready' && break
+        [[ $i -eq 4 ]] && fail "$HCLOUD_GW_NAME never finished cloud-init (ssh root@$HCLOUD_GW_ADDR, cloud-init status --long)"
+        sleep 60
    done
-    echo "  $node_src"
    ok

-    # Handover, and this list is the whole of it — the same four variables the
-    # vmtest phase sets. If it grows, the test is drifting away from production.
-    SITE=hcloud
+    step 0 "Closing egress (Hetzner firewall)..."
+    hcloud_apply_egress
+    ok
+
+    # Handover. SITE is homelab: the nodes take the production site patch
+    # unchanged; the WireGuard patches are the per-run addition.
+    SITE=homelab
    NODES=("${HCLOUD_NODES[@]}")
    WORKERS=()
    ALL_NODES=("${NODES[@]}")
@@ -803,9 +855,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
    # Isolation from the production cluster. Both halves are required: the
    # _PATH variables are derived before any phase runs, and the talos phase
    # writes to those.
-    HCLOUD_DIR=$(mktemp -d)
-    export TALOSCONFIG="$HCLOUD_DIR/talosconfig"
-    export KUBECONFIG="$HCLOUD_DIR/kubeconfig"
+    export TALOSCONFIG="$HCLOUD_STATE_DIR/talosconfig"
+    export KUBECONFIG="$HCLOUD_STATE_DIR/kubeconfig"
    TALOSCONFIG_PATH="$TALOSCONFIG"
    KUBECONFIG_PATH="$KUBECONFIG"

@@ -1132,9 +1183,38 @@ EOF
        || fail "the registry auth patch was not written to a readable file: '$REGISTRY_PATCH'"

    step 0 "Applying Talos configs to nodes..."
+    # The patch list every node gets, in order. NODE_PATCH_ARGS is the result.
+    node_patch_args() {
+        local node=$1
+        NODE_PATCH_ARGS=(
+            --config-patch "@$TALOS_DIR/patches/cluster.yaml"
+            --config-patch "@$TALOS_DIR/patches/site-${SITE}.yaml"
+            --config-patch "@$TALOS_DIR/patches/containerd-layer-retention.yaml"
+            --config-patch "@$TALOS_DIR/patches/${node}.yaml"
+            --config-patch "@$REGISTRY_PATCH"
+        )
+        # Per-run WireGuard keys and peers (hcloudtest only).
+        if [[ -n "${HCLOUD_WG_DIR:-}" ]]; then
+            NODE_PATCH_ARGS+=(--config-patch "@$HCLOUD_WG_DIR/${node}.yaml")
+        fi
+    }
+    # Re-applies a configured node without a reboot. Addressed by its
+    # InternalIP through the endpoint: once Cilium runs, apid does not answer
+    # requests addressed to the node's public IP.
+    reapply_node_config() {
+        local node=$1 base_config=$2 addr=$3 err target
+        node_patch_args "$node"
+        target=$(kubectl get node "$node" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' | awk '{print $1}')
+        [[ -n "$target" ]] || fail "$node has no InternalIP"
+        err=$(timeout 300 talosctl apply-config -n "$target" -e "$addr" --mode no-reboot \
+            --file "$base_config" "${NODE_PATCH_ARGS[@]}" 2>&1) \
+            || fail "re-applying $node ($addr) failed: ${err//$'\n'/ | }"
+        echo "  $node: re-applied"
+    }
    apply_node_config() {
        local node=$1 base_config=$2 fixed_addr=${3:-}
        local fqdn="${node}.${DOMAIN}"
+        node_patch_args "$node"
        echo "  ${fixed_addr:-$fqdn}:"
        for i in $(seq 1 60); do
            # A fixed address skips DNS. Real hardware has a name to resolve;
@@ -1150,12 +1230,7 @@ EOF
                continue
            fi
            if err=$(talosctl apply-config --insecure -n "$node_ip" \
-                --file "$base_config" \
-                --config-patch "@$TALOS_DIR/patches/cluster.yaml" \
-                --config-patch "@$TALOS_DIR/patches/site-${SITE}.yaml" \
-                --config-patch "@$TALOS_DIR/patches/containerd-layer-retention.yaml" \
-                --config-patch "@$TALOS_DIR/patches/${node}.yaml" \
-                --config-patch "@$REGISTRY_PATCH" 2>&1); then
+                --file "$base_config" "${NODE_PATCH_ARGS[@]}" 2>&1); then
                echo -e "    applied via $node_ip ${GREEN}(OK)${NC}"
                return 0
            fi
@@ -1321,6 +1396,66 @@ EOF
    grep -q 'policyMapMax' "$CILIUM_VALUES" \
        || fail "extracted Cilium values look wrong (no policyMapMax) — check $CILIUM_HR"

+    # Cilium's cni.confPath is a subdirectory of /etc/cni/net.d that Multus
+    # creates (#832). The kubelet sees /etc/cni read-only on Talos, so its
+    # DirectoryOrCreate cannot make it, and on an empty cluster Multus is not
+    # there yet: the agent would wait on the mount forever. A hostNetwork pod
+    # per node creates it from inside a container, as Multus does.
+    cni_conf_dir=$(yq -r '.cni.confPath' "$CILIUM_VALUES")
+    if [[ "$cni_conf_dir" == /etc/cni/net.d/* ]]; then
+        step 0 "Creating $cni_conf_dir on every node..."
+        # The pods need kube-system/default, which kube-controller-manager
+        # creates shortly after the API answers.
+        for i in $(seq 1 30); do
+            kubectl -n kube-system get serviceaccount default >/dev/null && break
+            sleep 5
+            [[ $i -eq 30 ]] && fail "kube-system/default ServiceAccount never appeared"
+        done
+        # A pod bound to a node that has not registered yet is garbage
+        # collected, and its wait below would never succeed.
+        for node in "${ALL_NODES[@]}"; do
+            for i in $(seq 1 60); do
+                kubectl get node "$node" >/dev/null && break
+                sleep 5
+                [[ $i -eq 60 ]] && fail "$node never registered with the API server"
+            done
+        done
+        for node in "${ALL_NODES[@]}"; do
+            kubectl apply -f - >/dev/null <<EOF || fail "could not create the cni-confdir pod on $node"
+apiVersion: v1
+kind: Pod
+metadata:
+  name: cni-confdir-$node
+  namespace: kube-system
+spec:
+  nodeName: $node
+  hostNetwork: true
+  restartPolicy: Never
+  tolerations:
+    - operator: Exists
+  containers:
+    - name: mkdir
+      image: docker.io/library/busybox:1.37
+      command: ["mkdir", "-p", "/host/${cni_conf_dir#/etc/cni/net.d/}"]
+      volumeMounts:
+        - name: netd
+          mountPath: /host
+  volumes:
+    - name: netd
+      hostPath:
+        path: /etc/cni/net.d
+        type: Directory
+EOF
+        done
+        for node in "${ALL_NODES[@]}"; do
+            kubectl -n kube-system wait --for=jsonpath='{.status.phase}'=Succeeded \
+                "pod/cni-confdir-$node" --timeout=5m >/dev/null \
+                || fail "creating $cni_conf_dir on $node did not succeed (kubectl -n kube-system describe pod cni-confdir-$node)"
+            kubectl -n kube-system delete pod "cni-confdir-$node" >/dev/null
+        done
+        ok
+    fi
+
    helm repo add cilium "$(yq -r 'select(.kind == "HelmRepository") | .spec.url' "$CILIUM_HR")" >/dev/null
    helm repo update cilium >/dev/null
    helm upgrade --install cilium cilium/cilium \
@@ -1332,6 +1467,17 @@ EOF
    rm -f "$CILIUM_VALUES"
    ok

+    # Multus is the only writer of /etc/cni/net.d (#832); Cilium's conflist sits
+    # in a subdirectory the runtime does not read. Without Multus the kubelet
+    # reports "cni plugin not initialized" and nothing schedules, Flux
+    # included. Applied from the same manifests Flux owns, which adopts them.
+    step 0 "Installing Multus (the CNI entry point)..."
+    kubectl apply -k "$REPO_ROOT/kubernetes/infrastructure/multus" >/dev/null \
+        || fail "could not apply kubernetes/infrastructure/multus"
+    kubectl -n kube-system rollout status daemonset/kube-multus-ds-subdir --timeout=10m >/dev/null \
+        || fail "Multus never became ready (kubectl -n kube-system get pods -l app=multus)"
+    ok
+
    step 0 "Waiting for K8s nodes..."
    expected=${#ALL_NODES[@]}
    # Before the loop, because 0 == 0 would satisfy its exit condition on the
@@ -1368,10 +1514,18 @@ EOF
        fi
    done

+    if [[ -n "${HCLOUD_WG_DIR:-}" ]]; then
+        hcloud_after_nodes_ready
+        hcloud_egress_check
+    fi
+
    # Stash configs for subsequent phases / export-access step. Skipped under
-    # vmtest: those are a throwaway cluster's credentials, and writing them here
-    # would leave the flux phase pointed at VMs that no longer exist.
-    if [[ "${VMTEST:-0}" == "1" ]]; then
+    # vmtest and hcloudtest: those are a throwaway cluster's credentials, and a
+    # later production `bootstrap.sh flux` copies access-staging/ over the
+    # operator's kubeconfig and talosconfig.
+    if [[ -n "${HCLOUD_WG_DIR:-}" ]]; then
+        echo -e "${GREEN}Talos phase complete.${NC} Access files in $HCLOUD_STATE_DIR/ (not staged)"
+    elif [[ "${VMTEST:-0}" == "1" ]]; then
        # Nodes being Ready says nothing about storage. The CephCluster picks
        # OSD nodes by a label out of the machine config and their device by the
        # partition label RawVolumeConfig creates, so assert both here — running
@@ -1620,7 +1774,9 @@ bring_up_openbao() {
    # entered by mistake. Wait for the container to be running first.
    echo -n "  waiting for ${pods[0]} to answer"
    local status="" status_err=""
-    for i in $(seq 1 90); do
+    # Bounded at 30 minutes: on a fresh cluster the pod waits for Ceph to
+    # provision its volume.
+    for i in $(seq 1 180); do
        if status=$(bao_exec "${pods[0]}" bao status -format=json 2>"$OPENBAO_SECRETS_DIR/.baoerr"); then
            echo; break
        fi
@@ -1638,7 +1794,7 @@ bring_up_openbao() {
        if jq -e 'has("initialized")' >/dev/null 2>&1 <<<"$status"; then echo; break; fi
        printf '.'
        sleep 10
-        if [[ $i -eq 90 ]]; then
+        if [[ $i -eq 180 ]]; then
            echo
            echo "  last error: $status_err" >&2
            fail "${pods[0]} never answered bao status — refusing to guess whether it is sealed or uninitialised"
@@ -1747,9 +1903,31 @@ if phase_runs flux; then
    case "$FLUX_SOURCE" in
        gitea)  GITEA_TOKEN=$(read_secret git-token);  export GITEA_TOKEN ;;
        github) GITHUB_TOKEN=$(read_secret git-token); export GITHUB_TOKEN ;;
-        *)      fail "Unknown FLUX_SOURCE '$FLUX_SOURCE' (want gitea|github)" ;;
+        restore-test)
+            GITHUB_TOKEN=$(read_secret git-token); export GITHUB_TOKEN
+            command -v gh >/dev/null || fail "gh not found (registers the read-only deploy key)"
+            # The throwaway cluster carries production secrets, and this phase
+            # copies access files over KUBECONFIG_PATH and TALOSCONFIG_PATH:
+            # neither may be the operator's production file.
+            [[ "$KUBECONFIG_PATH" != "$HOME/.kube/config" && "$TALOSCONFIG_PATH" != "$HOME/.talos/config" ]] \
+                || fail "FLUX_SOURCE=restore-test needs KUBECONFIG and TALOSCONFIG pointed at the test cluster ($HCLOUD_STATE_DIR)"
+            ;;
+        *)      fail "Unknown FLUX_SOURCE '$FLUX_SOURCE' (want gitea|github|restore-test)" ;;
    esac
-    [[ "$PHASE" == "flux" ]] && load_staged_access
+    # The restore test reads its access from KUBECONFIG/TALOSCONFIG, which
+    # hcloudtest writes; access-staging/ may hold an older run's files.
+    [[ "$PHASE" == "flux" && "$FLUX_SOURCE" != "restore-test" ]] && load_staged_access
+    if [[ "$FLUX_SOURCE" == "restore-test" ]]; then
+        # Whatever the paths say, the cluster itself must be the test cluster,
+        # before anything below writes a secret to it.
+        node_names=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}') \
+            || fail "cannot list the nodes of the target cluster"
+        [[ -n "$node_names" ]] || fail "the target cluster has no nodes"
+        for n in $node_names; do
+            [[ "$n" == "$HCLOUD_NAME_PREFIX"-* ]] \
+                || fail "node '$n' is not a $HCLOUD_NAME_PREFIX-* test node — refusing the restore test against this cluster"
+        done
+    fi

    echo -n "Creating SOPS age secret... "
    kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
@@ -1789,7 +1967,34 @@ if phase_runs flux; then
    fi
    unset seal_key existing

-    if [[ "$FLUX_SOURCE" == "github" ]]; then
+    if [[ "$FLUX_SOURCE" == "restore-test" ]]; then
+        # The production tree through kubernetes/restore-test/flux-root.yaml,
+        # whose patches are the test's whole deviation from production. Nothing
+        # is committed anywhere: the source is the GitHub mirror, read through a
+        # deploy key GitHub itself holds to read-only, so the test cluster
+        # cannot push (image automation) even with production's manifests.
+        step 0 "Bootstrapping FluxCD for the restore test (restore-test/flux-root.yaml)..."
+        flux install --components-extra=image-reflector-controller,image-automation-controller \
+            || fail "flux install failed"
+        restore_test_repo
+        deploy_key="$OPENBAO_SECRETS_DIR/restore-test-git"
+        ssh-keygen -q -t ed25519 -N '' -C "restore-test" -f "$deploy_key"
+        gh api -X POST "repos/$restore_repo/keys" \
+            -f title="restore-test $(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            -f key="$(cat "$deploy_key.pub")" -F read_only=true >/dev/null \
+            || fail "could not register the read-only deploy key on $restore_repo"
+        flux create secret git restore-test-git \
+            --url="$restore_url" \
+            --private-key-file="$deploy_key" >/dev/null \
+            || fail "could not create the restore-test-git Secret"
+        # On FLUX_BRANCH, so a branch can be tried here before it is merged.
+        yq -y --arg b "$FLUX_BRANCH" \
+            'if .kind == "GitRepository" and .metadata.name == "restore-test" then .spec.ref.branch = $b else . end' \
+            "$REPO_ROOT/kubernetes/restore-test/flux-root.yaml" | kubectl apply -f - >/dev/null \
+            || fail "could not apply restore-test/flux-root.yaml"
+        echo "  source: $restore_url, branch $FLUX_BRANCH"
+        ok
+    elif [[ "$FLUX_SOURCE" == "github" ]]; then
        step 0 "Bootstrapping FluxCD from the GitHub break-glass mirror..."
        echo -e "${YELLOW}BREAK-GLASS: Flux source is github.com/${GITHUB_OWNER}/${GITHUB_REPO}, NOT in-cluster Gitea.${NC}"
        echo -e "${YELLOW}Switch the source back to Gitea once it is serving again (see RUNBOOK-cold-rebuild.md).${NC}"
@@ -1893,7 +2098,8 @@ if phase_runs wait-apps; then
    echo -e "${YELLOW}=== Phase: Wait for apps ===${NC}"
    command -v kubectl >/dev/null || fail "kubectl not found"
    command -v flux >/dev/null    || fail "flux not found"
-    [[ "$PHASE" == "wait-apps" ]] && load_staged_access
+    # The restore test reads its access from KUBECONFIG/TALOSCONFIG.
+    [[ "$PHASE" == "wait-apps" && "$FLUX_SOURCE" != "restore-test" ]] && load_staged_access

    # Wait for a Flux Kustomization to become Ready=True. Prints a status line
    # whenever the Ready/Message pair changes, plus a heartbeat every 60 s so a
@@ -1940,8 +2146,8 @@ if phase_runs wait-apps; then
    # Each app KS now owns its CNPG database (+ any data PVCs) directly — the
    # per-app `seed-<app>` KSes that used to bootstrap them were folded into the
    # app KSes when the seed layer was retired. Each wait below gates on the app
-    # KS, whose readiness includes its CNPG cluster coming up (initdb on a fresh
-    # cluster; a cold-rebuild data restore is the separate RECOVERY.md runbook).
+    # KS, whose readiness includes its CNPG cluster coming up (bootstrap.recovery)
+    # and, through the restore KS it depends on, its volumes restored by Velero.
    # Timeouts stay generous to cover first-time cluster bootstrap on fresh Ceph.
    # apps waits on infrastructure-ceph-cluster (Ceph mon/mgr/OSD bring-up
    # is genuinely 5–10 min on a fresh OSD-wiped cluster).
@@ -17,6 +17,7 @@ spec:
  # woodpecker migrated) — secrets come from OpenBao via ESO
  # (MIGRATION-sops-openbao.md).
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -22,6 +22,7 @@ spec:
    secretRef:
      name: sops-age
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -20,6 +20,7 @@ spec:
  # from OpenBao via ESO (MIGRATION-sops-openbao.md). This was the LAST
  # sops consumer; phase 4 (age-key retirement) may proceed after soak.
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -14,7 +14,14 @@ spec:
  path: ./kubernetes/apps/gitea
  prune: true
  wait: true
+  # The claim's bound volume differs per cluster (apps/gitea/pvc.yaml).
+  ignore:
+    - paths: ["/spec/volumeName"]
+      target:
+        kind: PersistentVolumeClaim
+        name: data-gitea-0
  dependsOn:
+    - name: restore-gitea
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -25,6 +25,7 @@ spec:
  # decryption block dropped 2026-07-13: path is sops-free — secrets come
  # from OpenBao via ESO (MIGRATION-sops-openbao.md).
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -58,6 +59,7 @@ spec:
  prune: true
  wait: true
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -80,6 +82,7 @@ spec:
  # decryption block dropped 2026-07-13: path is sops-free — secrets come
  # from OpenBao via ESO (MIGRATION-sops-openbao.md).
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
 ---
@@ -120,6 +123,7 @@ spec:
  # First boot downloads the Whisper model (a few hundred MB) — give it room.
  wait: true
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -20,6 +20,7 @@ spec:
  # decryption block dropped 2026-07-13: path is sops-free — secrets come
  # from OpenBao via ESO (MIGRATION-sops-openbao.md).
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -25,6 +25,7 @@ spec:
  # boilerplate from the monolithic seed KS — InfluxDB has no CNPG cluster, only
  # a chart-provisioned ceph-block PVC, so ceph is the only storage dep needed.
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -17,6 +17,7 @@ spec:
  prune: true
  wait: true
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -22,6 +22,7 @@ spec:
    secretRef:
      name: sops-age
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -14,6 +14,7 @@ spec:
  prune: true
  wait: true
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -20,6 +20,7 @@ spec:
  # decryption block dropped 2026-07-13: path is sops-free — secrets come
  # from OpenBao via ESO (MIGRATION-sops-openbao.md).
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -0,0 +1,87 @@
+# Ready once Velero has completed a sync of the backups in the bucket. A
+# Restore created before that finds no backup and fails for good.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: velero-backups-synced
+  namespace: flux-system
+spec:
+  interval: 10m
+  timeout: 30m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/restore/gate
+  prune: true
+  dependsOn:
+    - name: infrastructure-crds
+  healthChecks:
+    - apiVersion: velero.io/v1
+      kind: BackupStorageLocation
+      name: default
+      namespace: velero
+  healthCheckExprs:
+    - apiVersion: velero.io/v1
+      kind: BackupStorageLocation
+      current: has(status.lastSyncedTime)
+---
+# Gitea's data first: it is the registry for the in-house images the rest of
+# the cluster pulls. The gitea Kustomization depends on this one only.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: restore-gitea
+  namespace: flux-system
+spec:
+  interval: 10m
+  timeout: 1h
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/restore/gitea
+  prune: true
+  wait: true
+  dependsOn:
+    - name: velero-backups-synced
+    - name: infrastructure-ceph-cluster
+  healthCheckExprs:
+    # A schedule without any backup has nothing to restore.
+    - apiVersion: velero.io/v1
+      kind: Restore
+      failed: >-
+        has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
+        (status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
+        status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
+      current: >-
+        has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
+---
+# App data: every other app Kustomization with backed-up volumes depends on
+# this one, so no app starts on an empty volume.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: restore
+  namespace: flux-system
+spec:
+  interval: 10m
+  timeout: 6h
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/restore
+  prune: true
+  wait: true
+  dependsOn:
+    - name: restore-gitea
+    - name: velero-backups-synced
+    - name: infrastructure-ceph-cluster
+  healthCheckExprs:
+    # A schedule without any backup has nothing to restore.
+    - apiVersion: velero.io/v1
+      kind: Restore
+      failed: >-
+        has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
+        (status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
+        status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
+      current: >-
+        has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
@@ -17,6 +17,7 @@ spec:
  prune: true
  wait: true
  dependsOn:
+    - name: restore
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
@@ -13,6 +13,7 @@ spec:
  path: ./kubernetes/apps/wordpress
  prune: true
  dependsOn:
+    - name: restore
    # infrastructure gates on the mariadb-operator HelmRelease being Ready,
    # so the MariaDB CRD exists and has a controller before this applies.
    - name: infrastructure
@@ -0,0 +1,449 @@
+# shellcheck shell=bash
+# Hetzner Cloud site for the hcloudtest phase of bootstrap.sh. Sourced, not run.
+#
+# Hetzner networks are layer 3 and IPv4-only, so the production VLAN trunk is
+# rebuilt as one WireGuard mesh per segment over a private underlay network:
+# the links carry the production names and addresses (talos/patches/hcloud-N.yaml),
+# and the server hcloud-gw stands in for the UDM on mgmt0 and ingress0 (DNS,
+# BGP peer, egress NAT). Keys, peers and the gateway's config are generated per
+# run into $OPENBAO_SECRETS_DIR and never written to the repo.
+
+HCLOUD_UNDERLAY_RANGE="10.90.0.0/16"
+# Outlives the run: hcloud-gw's ssh key, and the cluster's talosconfig and
+# kubeconfig, for inspecting the machines after the phase exits.
+HCLOUD_STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/homelab-hcloudtest"
+HCLOUD_GW_NAME="hcloud-gw"
+# Public IPv4 per node, filled in as the servers are created.
+declare -A HCLOUD_PUBLIC_IP=()
+# hcloud-gw's DNS upstreams: Hetzner's resolvers.
+HCLOUD_GW_DNS_UPSTREAMS=(185.12.64.1 185.12.64.2 2a01:4ff:ff00::add:1)
+HCLOUD_GW_UNDERLAY_IP="10.90.0.10"
+# hcloud-N gets 10.90.0.1N on the underlay.
+hcloud_underlay_ip() { printf '10.90.0.1%s' "${1##*-}"; }
+
+# Links that are WireGuard meshes, and the peers each one has besides the
+# gateway: "nodes" links reach the other nodes, "gw" links only the gateway.
+HCLOUD_WG_LINKS=(mgmt0 node0 pod0 ingress0)
+
+hcloud_link_port() {
+    yq -r --arg l "$1" 'select(.kind == "WireguardConfig" and .name == $l) | .listenPort' \
+        "$TALOS_DIR/patches/hcloud-1.yaml"
+}
+
+# Host routes for a node's addresses on one link, read from its patch.
+hcloud_link_hostroutes() {
+    local node=$1 link=$2
+    yq -r --arg l "$link" 'select(.kind == "WireguardConfig" and .name == $l) | .addresses[].address' \
+        "$TALOS_DIR/patches/$node.yaml" \
+        | awk -F/ '{ print $1 (index($1, ":") ? "/128" : "/32") }'
+}
+
+# The UDM's addresses, read from what the cluster config already states:
+# the egress gateway annotations (ingress0) and the resolver (mgmt0). The
+# resolver list also names k8s-gateway, which is a cluster Service, not the UDM.
+hcloud_gw_ingress_addrs() {
+    yq -r 'select(.machine) | .machine.nodeAnnotations["net.renner.uno/egress-gw-v4"]' "$TALOS_DIR/patches/site-homelab.yaml" | sed 's|$|/24|'
+    yq -r 'select(.machine) | .machine.nodeAnnotations["net.renner.uno/egress-gw-v6"]' "$TALOS_DIR/patches/site-homelab.yaml" | sed 's|$|/64|'
+}
+hcloud_gw_mgmt_hostroutes() {
+    local k8s_gateway
+    k8s_gateway=$(yq -r 'select(.kind == "HelmRelease") | .spec.values.service.clusterIP' \
+        "$REPO_ROOT/kubernetes/infrastructure/k8s-gateway/helmrelease.yaml")
+    [[ -n "$k8s_gateway" && "$k8s_gateway" != null ]] || fail "could not read the k8s-gateway ClusterIP"
+    yq -r --arg skip "$k8s_gateway" 'select(.kind == "ResolverConfig") | .nameservers[].address | select(. != $skip)' \
+        "$TALOS_DIR/patches/site-homelab.yaml" | sed 's|$|/32|'
+    yq -r 'select(.kind == "WireguardConfig" and .name == "mgmt0") | .routes[] | select(.destination) | .destination' "$TALOS_DIR/patches/hcloud-1.yaml"
+}
+# LoadBalancer pool, read from the CiliumLoadBalancerIPPool. Delivered to
+# hcloud-1 only: WireGuard maps a destination to exactly one peer.
+hcloud_lb_pool() {
+    local pool="$REPO_ROOT/kubernetes/infrastructure/post-crds/cilium-bgp.yaml"
+    yq -r 'select(.kind == "CiliumLoadBalancerIPPool") | .spec.blocks[] | .start' "$pool" \
+        | awk '{ print $1 (index($1, ":") ? "/120" : "/24") }'
+}
+
+# One keypair per machine and link.
+hcloud_wg_keys() {
+    local m l
+    for m in "$HCLOUD_GW_NAME" "${HCLOUD_NODES[@]}"; do
+        for l in "${HCLOUD_WG_LINKS[@]}"; do
+            [[ -s "$OPENBAO_SECRETS_DIR/wg-$m-$l.key" ]] && continue
+            ( umask 077; wg genkey > "$OPENBAO_SECRETS_DIR/wg-$m-$l.key" )
+            wg pubkey < "$OPENBAO_SECRETS_DIR/wg-$m-$l.key" > "$OPENBAO_SECRETS_DIR/wg-$m-$l.pub"
+        done
+    done
+}
+hcloud_wg_pub() { cat "$OPENBAO_SECRETS_DIR/wg-$1-$2.pub"; }
+hcloud_wg_priv() { cat "$OPENBAO_SECRETS_DIR/wg-$1-$2.key"; }
+
+# Pod CIDRs per node, once the cluster has allocated them. Empty before.
+hcloud_pod_cidrs() {
+    [[ -s "$OPENBAO_SECRETS_DIR/podcidrs" ]] || return 0
+    awk -v n="$1" '$1 == n { for (i = 2; i <= NF; i++) print $i }' "$OPENBAO_SECRETS_DIR/podcidrs"
+}
+
+# Every source a node can send from: its addresses on all links and its pod
+# CIDRs. A VLAN accepts any source; WireGuard only those in the peer's
+# allowedIPs, and the kernel picks the egress link independently of the
+# source (Cilium SNATs to pod0's address and routes via node0).
+hcloud_peer_sources() {
+    local node=$1 l
+    for l in "${HCLOUD_WG_LINKS[@]}"; do
+        hcloud_link_hostroutes "$node" "$l"
+    done
+    hcloud_pod_cidrs "$node"
+}
+
+# Writes $HCLOUD_WG_DIR/<node>.yaml: private keys and peers for every link.
+hcloud_node_wg_patch() {
+    local node=$1 out="$HCLOUD_WG_DIR/$1.yaml" l port peer first=1
+    {
+        for l in "${HCLOUD_WG_LINKS[@]}"; do
+            port=$(hcloud_link_port "$l")
+            [[ $first -eq 1 ]] || echo "---"
+            first=0
+            cat <<EOF
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: $l
+privateKey: $(hcloud_wg_priv "$node" "$l")
+peers:
+EOF
+            case "$l" in
+                mgmt0|ingress0)
+                    echo "  - publicKey: $(hcloud_wg_pub "$HCLOUD_GW_NAME" "$l")"
+                    echo "    endpoint: $HCLOUD_GW_UNDERLAY_IP:$port"
+                    echo "    persistentKeepaliveInterval: 25s"
+                    echo "    allowedIPs:"
+                    echo "      - 0.0.0.0/0"
+                    echo "      - ::/0"
+                    ;;
+            esac
+            if [[ "$l" != ingress0 ]]; then
+                for peer in "${HCLOUD_NODES[@]}"; do
+                    [[ "$peer" == "$node" ]] && continue
+                    echo "  - publicKey: $(hcloud_wg_pub "$peer" "$l")"
+                    echo "    endpoint: $(hcloud_underlay_ip "$peer"):$port"
+                    echo "    persistentKeepaliveInterval: 25s"
+                    echo "    allowedIPs:"
+                    hcloud_peer_sources "$peer" | sed 's/^/      - /'
+                done
+            fi
+        done
+        # The default route is mgmt0's; replies from the public address
+        # (talosctl and kubectl from the operator) leave through enp1s0.
+        if [[ -n "${HCLOUD_PUBLIC_IP[$node]:-}" ]]; then
+            cat <<EOF
+---
+apiVersion: v1alpha1
+kind: RoutingRuleConfig
+name: "100"
+src: ${HCLOUD_PUBLIC_IP[$node]}/32
+table: "100"
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+routes:
+  - destination: 172.31.1.1/32
+    table: "100"
+  - gateway: 172.31.1.1
+    table: "100"
+EOF
+        fi
+    } > "$out"
+}
+
+# The gateway's peer list for one link, as `wg set` arguments.
+hcloud_gw_peer_args() {
+    local l=$1 node ips
+    for node in "${HCLOUD_NODES[@]}"; do
+        ips=$(hcloud_peer_sources "$node")
+        if [[ "$l" == ingress0 && "$node" == "${HCLOUD_NODES[0]}" ]]; then
+            ips+=$'\n'$(hcloud_lb_pool)
+        fi
+        ips=$(grep -v '^$' <<<"$ips" | paste -sd, -)
+        printf 'peer %s endpoint %s:%s persistent-keepalive 25 allowed-ips %s\n' \
+            "$(hcloud_wg_pub "$node" "$l")" "$(hcloud_underlay_ip "$node")" "$(hcloud_link_port "$l")" "$ips"
+    done
+}
+
+# cloud-init for the gateway, written to $OPENBAO_SECRETS_DIR/gw-userdata.yaml.
+# Brings up mgmt0 and ingress0 with the UDM's addresses, NATs egress, answers
+# DNS and peers BGP with every node. Called directly, not in $(...), so that
+# fail stops the run.
+hcloud_gw_userdata() {
+    local out="$OPENBAO_SECRETS_DIR/gw-userdata.yaml" l addrs node n a
+    local gw4 gw6 ingress
+    mapfile -t ingress < <(hcloud_gw_ingress_addrs)
+    gw4=${ingress[0]:-}; gw4=${gw4%/*}
+    gw6=${ingress[1]:-}; gw6=${gw6%/*}
+    [[ -n "$gw4" && "$gw4" != null && -n "$gw6" && "$gw6" != null ]] || fail "could not read the gateway's ingress0 addresses from site-homelab.yaml"
+    {
+        echo "#cloud-config"
+        echo "package_update: true"
+        echo "packages: [wireguard-tools, bird2, dnsmasq, nftables]"
+        echo "write_files:"
+        for l in mgmt0 ingress0; do
+            case "$l" in
+                mgmt0)    addrs=$(hcloud_gw_mgmt_hostroutes | sed -E 's|/32$|/24|'
+                          printf 'fd40::1/64\nfd02:40::1/64\n') ;;
+                ingress0) addrs=$(hcloud_gw_ingress_addrs) ;;
+            esac
+            echo "  - path: /etc/wireguard/$l.key"
+            echo "    permissions: '0600'"
+            echo "    content: $(hcloud_wg_priv "$HCLOUD_GW_NAME" "$l")"
+            echo "  - path: /etc/wireguard/$l.peers"
+            echo "    permissions: '0600'"
+            echo "    content: |"
+            hcloud_gw_peer_args "$l" | sed 's/^/      /'
+            echo "  - path: /etc/wireguard/$l.addrs"
+            echo "    content: |"
+            sed 's/^/      /' <<<"$addrs"
+        done
+        cat <<'EOF'
+  - path: /usr/local/sbin/hcloud-gw-link
+    permissions: '0755'
+    content: |
+      #!/bin/sh
+      # hcloud-gw-link <link> <port>: (re)creates a WireGuard link from /etc/wireguard.
+      set -eu
+      l=$1; port=$2
+      [ -e "/sys/class/net/$l" ] || ip link add "$l" type wireguard
+      wg set "$l" listen-port "$port" private-key "/etc/wireguard/$l.key"
+      while read -r line; do [ -n "$line" ] && wg set "$l" $line; done < "/etc/wireguard/$l.peers"
+      ip link set "$l" mtu 1370 up
+      while read -r a; do [ -n "$a" ] && ip addr replace "$a" dev "$l"; done < "/etc/wireguard/$l.addrs"
+      # Pod CIDRs and the LB pool route via ingress0, as the UDM learns them
+      # by BGP. Node host addresses (/32, /128) are reached through each
+      # link's connected subnet instead.
+      [ "$l" = ingress0 ] || exit 0
+      for a in $(wg show "$l" allowed-ips | cut -f2-); do
+        case "$a" in 0.0.0.0/0|::/0|"(none)"|*/32|*/128) ;; *) ip route replace "$a" dev "$l" proto static ;; esac
+      done
+  - path: /etc/sysctl.d/90-gw.conf
+    content: |
+      net.ipv4.ip_forward=1
+      net.ipv6.conf.all.forwarding=1
+      net.ipv6.conf.eth0.accept_ra=2
+  - path: /etc/nftables.conf
+    content: |
+      flush ruleset
+      table inet nat {
+        chain postrouting {
+          type nat hook postrouting priority srcnat;
+          oifname "eth0" ip saddr 10.0.0.0/8 masquerade
+          oifname "eth0" ip6 saddr fd00::/8 masquerade
+        }
+      }
+  - path: /etc/dnsmasq.d/gw.conf
+    content: |
+      bind-dynamic
+      interface=mgmt0
+      except-interface=lo
+      no-resolv
+      # renner.uno is answered locally only: the cluster's k8s-gateway serves
+      # its own names; the public zone's wildcard leads to production.
+      server=/renner.uno/
+EOF
+        printf '      server=%s\n' "${HCLOUD_GW_DNS_UPSTREAMS[@]}"
+        echo "  - path: /etc/bird/bird.conf"
+        echo "    content: |"
+        {
+            echo "router id $gw4;"
+            echo "protocol device { scan time 10; }"
+            echo "protocol kernel { ipv4 { import none; export all; }; merge paths on; }"
+            echo "protocol kernel { ipv6 { import none; export all; }; merge paths on; }"
+            echo "template bgp n4 { local $gw4 as 64512; neighbor as 64513; direct; ipv4 { import all; export none; }; }"
+            echo "template bgp n6 { local $gw6 as 64512; neighbor as 64513; direct; ipv6 { import all; export none; }; }"
+            for node in "${HCLOUD_NODES[@]}"; do
+                n=${node//-/_}
+                while read -r a; do
+                    a=${a%/*}
+                    if [[ "$a" == *:* ]]; then
+                        echo "protocol bgp ${n}_v6 from n6 { neighbor $a; }"
+                    else
+                        echo "protocol bgp ${n}_v4 from n4 { neighbor $a; }"
+                    fi
+                done < <(hcloud_link_hostroutes "$node" ingress0)
+            done
+        } | sed 's/^/      /'
+        # One command: the ready marker exists only if every step succeeded.
+        echo "runcmd:"
+        echo "  - [ sh, -c, 'sysctl --system && /usr/local/sbin/hcloud-gw-link mgmt0 $(hcloud_link_port mgmt0) && /usr/local/sbin/hcloud-gw-link ingress0 $(hcloud_link_port ingress0) && systemctl enable --now nftables && systemctl restart dnsmasq && systemctl restart bird && touch /run/hcloud-gw-ready' ]"
+    } > "$out"
+}
+
+# One multiplexed connection per run: Hetzner filters a source that opens many
+# ssh connections in a short time. The socket is per process, so a gateway
+# recreated under the same address never meets an older master; on_exit
+# closes it.
+hcloud_gw_ssh() {
+    ssh -i "$HCLOUD_STATE_DIR/gw-ssh" -o BatchMode=yes -o ConnectTimeout=15 \
+        -o ControlMaster=auto -o ControlPath="$(hcloud_gw_control_path)" -o ControlPersist=30m \
+        -o UserKnownHostsFile="$HCLOUD_STATE_DIR/gw-known-hosts" -o StrictHostKeyChecking=accept-new \
+        "root@$HCLOUD_GW_ADDR" "$@"
+}
+hcloud_gw_control_path() { printf '%s/hcgw-%s' "${XDG_RUNTIME_DIR:-/tmp}" "$$"; }
+hcloud_gw_ssh_close() {
+    [[ -S "$(hcloud_gw_control_path)" ]] || return 0
+    ssh -o ControlPath="$(hcloud_gw_control_path)" -O exit "root@$HCLOUD_GW_ADDR" \
+        || echo "  WARNING: ssh master $(hcloud_gw_control_path) not closed" >&2
+}
+
+# The Talos snapshot this cluster installs from, into HCLOUD_IMAGE. Built when
+# missing: a throwaway server in rescue mode writes the Image Factory hcloud
+# image to its disk, which is then snapshotted. Matched by version and
+# schematic. Called directly, not in $(...), so that fail stops the run.
+hcloud_talos_snapshot() {
+    local labels="os=talos,talos-version=$VM_VERSION,talos-schematic=${VM_SCHEMATIC:0:16}"
+    local builder="$HCLOUD_NAME_PREFIX-imgbuild" key_dir addr ids listing
+    listing=$(hcloud image list --type snapshot -l "$labels" -o noheader -o columns=id) \
+        || fail "could not list the Talos snapshots"
+    mapfile -t ids <<<"$listing"
+    if [[ -n "${ids[0]:-}" ]]; then
+        HCLOUD_IMAGE=${ids[0]}
+        return 0
+    fi
+    echo "  no snapshot for $VM_VERSION/${VM_SCHEMATIC:0:16}, building one"
+    key_dir=$(mktemp -d)
+    ssh-keygen -q -t ed25519 -N '' -C "$builder" -f "$key_dir/key"
+    if hcloud ssh-key list -l "$HCLOUD_LABEL" -o noheader -o columns=name | grep -qx "$builder"; then
+        hcloud ssh-key delete "$builder" >/dev/null || fail "could not replace the stale image builder key"
+    fi
+    hcloud ssh-key create --name "$builder" --label "$HCLOUD_LABEL" \
+        --public-key-from-file "$key_dir/key.pub" >/dev/null || fail "could not upload the image builder key"
+    hcloud server create --name "$builder" --type cx23 --image debian-12 \
+        --location "${HCLOUD_LOCATION:-fsn1}" --ssh-key "$builder" --label "$HCLOUD_LABEL" >/dev/null \
+        || fail "could not create the image builder"
+    # From here a failure leaves a billed server: remove it before failing.
+    builder_fail() {
+        hcloud server delete "$builder" >/dev/null || echo "  WARNING: $builder not deleted" >&2
+        hcloud ssh-key delete "$builder" >/dev/null || echo "  WARNING: ssh key $builder not deleted" >&2
+        rm -rf "$key_dir"
+        fail "$1"
+    }
+    hcloud server enable-rescue "$builder" --ssh-key "$builder" >/dev/null || builder_fail "could not enable rescue on $builder"
+    hcloud server reset "$builder" >/dev/null || builder_fail "could not reset $builder into rescue"
+    addr=$(hcloud server ip "$builder") || builder_fail "could not read the address of $builder"
+    for i in $(seq 1 30); do
+        ssh -i "$key_dir/key" -o BatchMode=yes -o ConnectTimeout=10 \
+            -o UserKnownHostsFile="$key_dir/known_hosts" -o StrictHostKeyChecking=accept-new \
+            "root@$addr" 'grep -q rescue /etc/hostname || hostname | grep -q rescue' && break
+        sleep 10
+        [[ $i -eq 30 ]] && builder_fail "$builder never came up in rescue"
+    done
+    ssh -i "$key_dir/key" -o BatchMode=yes -o UserKnownHostsFile="$key_dir/known_hosts" "root@$addr" \
+        "set -eo pipefail; curl -fsSL https://factory.talos.dev/image/$VM_SCHEMATIC/$VM_VERSION/hcloud-amd64.raw.xz | xz -d | dd of=/dev/sda bs=4M conv=fsync" \
+        || builder_fail "writing the Talos image on $builder failed"
+    hcloud server poweroff "$builder" >/dev/null || builder_fail "could not power off $builder"
+    hcloud server create-image "$builder" --type snapshot \
+        --description "talos $VM_VERSION ${VM_SCHEMATIC:0:16}" --label "$labels" \
+        || builder_fail "could not snapshot $builder"
+    listing=$(hcloud image list --type snapshot -l "$labels" -o noheader -o columns=id) \
+        || builder_fail "could not list the Talos snapshots"
+    mapfile -t ids <<<"$listing"
+    [[ -n "${ids[0]:-}" ]] || builder_fail "the snapshot of $builder is not listed under $labels"
+    HCLOUD_IMAGE=${ids[0]}
+    hcloud server delete "$builder" >/dev/null || echo "  WARNING: $builder not deleted" >&2
+    hcloud ssh-key delete "$builder" >/dev/null || echo "  WARNING: ssh key $builder not deleted" >&2
+    rm -rf "$key_dir"
+}
+
+
+# Egress: ports only. hcloud-gw carries all pod traffic, the nodes only host
+# processes (image pulls); both get HTTPS, NTP and DNS to the gateway's
+# upstreams, the gateway also ssh (the Flux source).
+# This run's firewalls are attached before an earlier run's are removed, so no
+# machine is ever without one.
+hcloud_apply_egress() {
+    local gw_fw="hcloudtest-gw-egress-$$" node_fw="hcloudtest-node-egress-$$" fw proto resolvers
+    local fw_id fw_name srv_id
+    resolvers=$(printf '%s\n' "${HCLOUD_GW_DNS_UPSTREAMS[@]}" \
+        | awk '{ print $1 (index($1, ":") ? "/128" : "/32") }' | paste -sd, -)
+    for fw in "$gw_fw" "$node_fw"; do
+        hcloud firewall create --name "$fw" --label "$HCLOUD_LABEL" >/dev/null || fail "could not create $fw"
+        hcloud firewall add-rule "$fw" --direction out --protocol tcp --port 443 \
+            --destination-ips 0.0.0.0/0,::/0 --description "HTTPS" >/dev/null || fail "could not add HTTPS to $fw"
+        hcloud firewall add-rule "$fw" --direction out --protocol udp --port 123 \
+            --destination-ips 0.0.0.0/0,::/0 --description "NTP" >/dev/null || fail "could not add NTP to $fw"
+        for proto in udp tcp; do
+            hcloud firewall add-rule "$fw" --direction out --protocol "$proto" --port 53 \
+                --destination-ips "$resolvers" --description "DNS" >/dev/null || fail "could not add DNS to $fw"
+        done
+    done
+    # Not GitHub's published git ranges: github.com resolves to addresses
+    # outside them.
+    hcloud firewall add-rule "$gw_fw" --direction out --protocol tcp --port 22 \
+        --destination-ips 0.0.0.0/0,::/0 --description "ssh (Flux source)" >/dev/null \
+        || fail "could not add ssh to $gw_fw"
+    hcloud firewall apply-to-resource "$gw_fw" --type server --server "$HCLOUD_GW_NAME" >/dev/null \
+        || fail "could not attach $gw_fw"
+    for fw in "${HCLOUD_NODES[@]}"; do
+        hcloud firewall apply-to-resource "$node_fw" --type server --server "$fw" >/dev/null \
+            || fail "could not attach $node_fw to $fw"
+    done
+    while read -r fw_id fw_name; do
+        [[ -n "$fw_id" && "$fw_name" == hcloudtest-*-egress-* ]] || continue
+        [[ "$fw_name" == "$gw_fw" || "$fw_name" == "$node_fw" ]] && continue
+        while read -r srv_id; do
+            [[ -n "$srv_id" ]] || continue
+            hcloud firewall remove-from-resource "$fw_id" --type server --server "$srv_id" >/dev/null \
+                || fail "could not detach $fw_name from server $srv_id"
+        done < <(hcloud firewall describe "$fw_id" -o json | jq -r '.applied_to[] | select(.type == "server") | .server.id')
+        hcloud firewall delete "$fw_id" >/dev/null || fail "could not delete the previous firewall $fw_name"
+        echo "  removed $fw_name"
+    done < <(hcloud firewall list -l "$HCLOUD_LABEL" -o noheader -o columns=id,name)
+    echo "  $HCLOUD_GW_NAME: HTTPS, ssh, DNS, NTP; nodes: HTTPS, DNS, NTP"
+}
+
+# Runs restore-test/egress-check.yaml and fails unless every expectation holds.
+hcloud_egress_check() {
+    step 0 "Checking egress containment from a pod..."
+    kubectl delete pod egress-check --ignore-not-found --wait=true >/dev/null
+    sed "s|GW_PUBLIC_IP_PLACEHOLDER|$HCLOUD_GW_ADDR|" "$REPO_ROOT/kubernetes/restore-test/egress-check.yaml" \
+        | kubectl apply -f - >/dev/null || fail "could not start the egress check"
+    local phase=""
+    for i in $(seq 1 48); do
+        phase=$(kubectl get pod egress-check -o jsonpath='{.status.phase}') || phase=""
+        [[ "$phase" == Succeeded || "$phase" == Failed ]] && break
+        sleep 5
+    done
+    if [[ "$phase" != Succeeded && "$phase" != Failed ]]; then
+        kubectl describe pod egress-check | sed 's/^/  /' || true
+        fail "the egress check never finished (phase '${phase:-none}') — do not start Flux"
+    fi
+    kubectl logs egress-check | sed 's/^/  /' || true
+    [[ "$phase" == "Succeeded" ]] || fail "egress containment does not hold — do not start Flux"
+    kubectl delete pod egress-check --wait=false >/dev/null
+    ok
+}
+
+# After the nodes are Ready: the cluster has allocated pod CIDRs, which the
+# node0/pod0 meshes and the gateway's ingress0 have to route. Re-applies every
+# node's WireGuard patch without a reboot and updates the gateway in place.
+hcloud_after_nodes_ready() {
+    local node cidrs l
+    step 0 "Routing pod CIDRs over the WireGuard meshes..."
+    : > "$OPENBAO_SECRETS_DIR/podcidrs"
+    for node in "${HCLOUD_NODES[@]}"; do
+        cidrs=$(kubectl get node "$node" -o jsonpath='{.spec.podCIDRs[*]}')
+        [[ -n "$cidrs" ]] || fail "$node has no pod CIDRs allocated"
+        echo "$node $cidrs" >> "$OPENBAO_SECRETS_DIR/podcidrs"
+        echo "  $node: $cidrs"
+    done
+    for node in "${HCLOUD_NODES[@]}"; do
+        hcloud_node_wg_patch "$node"
+    done
+    for idx in "${!ALL_NODES[@]}"; do
+        reapply_node_config "${ALL_NODES[$idx]}" "$TALOS_GEN_DIR/controlplane.yaml" "${NODE_ADDRS[$idx]}"
+    done
+    for l in mgmt0 ingress0; do
+        hcloud_gw_peer_args "$l" > "$OPENBAO_SECRETS_DIR/gw-$l.peers"
+        hcloud_gw_ssh "cat > /etc/wireguard/$l.peers" < "$OPENBAO_SECRETS_DIR/gw-$l.peers" \
+            || fail "could not update the gateway's $l peers"
+        hcloud_gw_ssh "/usr/local/sbin/hcloud-gw-link $l $(hcloud_link_port "$l")" \
+            || fail "could not apply the gateway's $l peers"
+    done
+    ok
+}
@@ -0,0 +1,79 @@
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: ceph-csi-operator
+  namespace: rook-ceph
+spec:
+  interval: 24h
+  url: https://ceph.github.io/ceph-csi-operator
+---
+# The CSI Driver and OperatorConfig resources Rook requires
+# (rook/Documentation/Helm-Charts/csi-drivers-chart.md). Values are Rook's
+# recommended ones (rook/deploy/charts/ceph-csi-drivers/values.yaml) plus the
+# shape this cluster runs with.
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: ceph-csi-drivers
+  namespace: rook-ceph
+spec:
+  interval: 30m
+  dependsOn:
+    - name: rook-ceph-operator
+  chart:
+    spec:
+      chart: ceph-csi-drivers
+      sourceRef:
+        kind: HelmRepository
+        name: ceph-csi-operator
+        namespace: rook-ceph
+  install:
+    remediation:
+      retries: 3
+  upgrade:
+    remediation:
+      retries: 3
+  values:
+    operatorConfig:
+      namespace: rook-ceph
+      driverSpecDefaults:
+        imageSet:
+          name: rook-csi-operator-image-set-configmap
+        nodePlugin:
+          priorityClassName: system-node-critical
+        controllerPlugin:
+          priorityClassName: system-cluster-critical
+    drivers:
+      rbd:
+        enabled: true
+        name: rook-ceph.rbd.csi.ceph.com
+        # Velero snapshots ceph-block claims through CSI.
+        snapshotPolicy: volumeSnapshot
+        nodePlugin:
+          tolerations:
+            - key: workload-class
+              operator: Equal
+              value: pinned
+              effect: NoSchedule
+        controllerPlugin:
+          replicas: 2
+          deploymentStrategy:
+            type: Recreate
+      cephfs:
+        enabled: true
+        name: rook-ceph.cephfs.csi.ceph.com
+        snapshotPolicy: volumeGroupSnapshot
+        nodePlugin:
+          tolerations:
+            - key: workload-class
+              operator: Equal
+              value: pinned
+              effect: NoSchedule
+        controllerPlugin:
+          replicas: 2
+          deploymentStrategy:
+            type: Recreate
+      nfs:
+        enabled: false
+      nvmeof:
+        enabled: false
@@ -1,5 +1,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - helmrelease.yaml
  - serviceaccounts.yaml
  - rbac.yaml
@@ -9,7 +9,8 @@
 # snapshotMoveData uploads PVC contents through Kopia and the node-agent,
 # independently of the K8s API surface.
 #
-# Naming: <app>-daily. Restore is manual — see RUNBOOK-cold-rebuild.md.
+# Naming: <app>-daily. Each has a Restore in kubernetes/restore/, which a fresh
+# cluster runs before the app starts.
 # Retention matches the CNPG Barman policy.

 apiVersion: velero.io/v1
@@ -199,7 +200,6 @@ spec:
  # This is the ONLY restore path for home-assistant-config (.storage,
  # secrets.yaml, the config git checkout). The HAOS-backup `seed`
  # initContainer that used to repopulate it was removed in 21a0a43f.
-  # Restore procedure and test: REN-249 runbook section C.2.
  schedule: "45 6 * * *"
  useOwnerReferencesInBackup: false
  template:
@@ -0,0 +1,35 @@
+# Asserts the restore test's containment from a pod, before Flux restores
+# production's secrets. Applied by bootstrap.sh hcloudtest; exits non-zero on
+# the first expectation that does not hold.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: egress-check
+  namespace: default
+spec:
+  restartPolicy: Never
+  containers:
+    - name: check
+      image: docker.io/nicolaka/netshoot:v0.13
+      command:
+        - /bin/bash
+        - -c
+        - |
+          rc=0
+          expect() { # <description> <expected> <actual>
+            if [[ "$3" == $2 ]]; then echo "PASS $1: $3"; else echo "FAIL $1: got '$3', want '$2'"; rc=1; fi
+          }
+          code() { curl -sS -m 8 -o /dev/null -w '%{http_code} via %{remote_ip}' "$1" 2>/dev/null || true; }
+          expect "chart release download reachable" "302 via *" "$(code https://github.com/vmware-tanzu/helm-charts/releases/download/velero-11.3.2/velero-11.3.2.tgz)"
+          expect "smtp blocked" "closed" "$(nc -z -w 5 smtp.gmail.com 587 && echo open || echo closed)"
+          expect "Pages chart index reachable" "200 via *" "$(code https://prometheus-community.github.io/helm-charts/index.yaml)"
+          expect "ghcr.io reachable" "401 via *" "$(code https://ghcr.io/v2/)"
+          expect "object storage reachable" "[1-4]?? via *" "$(code https://fsn1.your-objectstorage.com)"
+          expect "github.com ssh reachable" "open" "$(nc -z -w 5 github.com 22 && echo open || echo closed)"
+          expect "gitea.renner.uno unresolved" "" "$(dig +short gitea.renner.uno)"
+          expect "public DNS resolves" "?*" "$(dig +short example.com A | head -1)"
+          expect "egress leaves through hcloud-gw" "$GW_PUBLIC_IP" "$(curl -4 -sS -m 8 https://ifconfig.me 2>/dev/null || true)"
+          exit $rc
+      env:
+        - name: GW_PUBLIC_IP
+          value: GW_PUBLIC_IP_PLACEHOLDER
@@ -0,0 +1,147 @@
+# Flux entry point of the off-site restore test (bootstrap.sh hcloudtest, then
+# FLUX_SOURCE=restore-test bootstrap.sh flux). Not reconciled by production.
+#
+# Applies the production tree unchanged. spec.patches below is the complete
+# list of differences between the test cluster and production.
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: restore-test
+  namespace: flux-system
+spec:
+  interval: 10m
+  url: ssh://git@github.com/konairius/Homelab.git
+  ref:
+    branch: master
+  # Read-only deploy key, created per run by bootstrap.sh.
+  secretRef:
+    name: restore-test-git
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: restore-test
+  namespace: flux-system
+spec:
+  interval: 10m
+  path: ./kubernetes/clusters/homelab
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: restore-test
+  patches:
+    # 1. Source: the GitHub mirror, read-only. In-cluster Gitea is restored by
+    #    this test, not available to it.
+    - target:
+        kind: GitRepository
+        name: flux-system
+        namespace: flux-system
+      patch: |
+        - op: replace
+          path: /spec/url
+          value: ssh://git@github.com/konairius/Homelab.git
+        - op: replace
+          path: /spec/secretRef/name
+          value: restore-test-git
+    # 2. The tree's own root sync: suspended, this Kustomization replaces it.
+    - target:
+        kind: Kustomization
+        name: flux-system
+        namespace: flux-system
+      patch: |
+        - op: add
+          path: /spec/suspend
+          value: true
+    # 3. Every Kustomization of the tree reads restore-test (and so the
+    #    branch bootstrap.sh sets on it), and carries the patches below; a
+    #    target a Kustomization does not contain matches nothing.
+    - target:
+        kind: Kustomization
+      patch: |
+        - op: replace
+          path: /spec/sourceRef/name
+          value: restore-test
+        - op: add
+          path: /spec/patches
+          value:
+            # 3a. No Ceph cluster network. Hetzner Cloud networks carry no
+            #     layer 2, so the ceph-san attachment cannot reach other
+            #     nodes; Ceph replicates over the public (pod) network.
+            - target:
+                kind: CephCluster
+                name: rook-ceph
+              patch: |
+                - op: remove
+                  path: /spec/network
+            # 3b-3f. No backups, retention or maintenance against the
+            #     production buckets: the test cluster holds production's
+            #     credentials. Velero restores still write their records.
+            # 3b. Velero: ReadOnly blocks backups, deletion, GC and repository
+            #     maintenance; restores and the backup sync still work.
+            - target:
+                kind: HelmRelease
+                name: velero
+              patch: |
+                - op: add
+                  path: /spec/values/configuration/backupStorageLocation/0/accessMode
+                  value: ReadOnly
+            # 3c. CNPG: no WAL archiving. The plugin stays loaded for recovery.
+            - target:
+                group: postgresql.cnpg.io
+                kind: Cluster
+              patch: |
+                - op: add
+                  path: /spec/plugins/0/isWALArchiver
+                  value: false
+            # 3d. CNPG: no retention. The plugin sidecar deletes by
+            #     retentionPolicy whether or not it archives.
+            - target:
+                group: barmancloud.cnpg.io
+                kind: ObjectStore
+              patch: |
+                apiVersion: barmancloud.cnpg.io/v1
+                kind: ObjectStore
+                metadata:
+                  name: _
+                spec:
+                  retentionPolicy: null
+            # 3e. CNPG: no base backups.
+            - target:
+                group: postgresql.cnpg.io
+                kind: ScheduledBackup
+              patch: |
+                - op: add
+                  path: /spec/suspend
+                  value: true
+            # 3g. Credential rotators: they roll production's tokens at
+            #     their issuers (Cloudflare, Authentik, Claude, ...).
+            - target:
+                group: batch
+                kind: CronJob
+                labelSelector: app.kubernetes.io/component=secret-rotator
+              patch: |
+                - op: add
+                  path: /spec/suspend
+                  value: true
+            # 3i. Gitea: no egress beyond the cluster. Its restored push
+            #     mirror force-pushes (--mirror) the backup's refs over
+            #     the GitHub mirror.
+            - target:
+                kind: CiliumNetworkPolicy
+                name: default-deny
+                namespace: gitea
+              patch: |
+                - op: test
+                  path: /spec/egress/5/toEntities
+                  value: [host, remote-node, world]
+                - op: replace
+                  path: /spec/egress/5/toEntities
+                  value: [host, remote-node]
+            # 3f. MariaDB: no logical dumps.
+            - target:
+                group: k8s.mariadb.com
+                kind: Backup
+              patch: |
+                - op: replace
+                  path: /spec/schedule/suspend
+                  value: true
@@ -0,0 +1,5 @@
+# Applies nothing: its Kustomization exists for its health check on the
+# BackupStorageLocation (clusters/homelab/restore.yaml).
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources: []
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - restore.yaml
@@ -0,0 +1,8 @@
+# Gitea first: it is the registry for the in-house images.
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: gitea
+  namespace: velero
+spec:
+  scheduleName: gitea-daily
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - restores.yaml
@@ -0,0 +1,99 @@
+# One Restore per Velero Schedule (infrastructure/post-crds/velero-schedules.yaml),
+# from its newest completed backup. A claim that already exists is skipped.
+# openbao-daily has none: bootstrap.sh restores OpenBao from its raft snapshot.
+# gitea-daily is restored first, by restore-gitea (gitea/restore.yaml).
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: paperless
+  namespace: velero
+spec:
+  scheduleName: paperless-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: audiobookshelf
+  namespace: velero
+spec:
+  scheduleName: audiobookshelf-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: authentik
+  namespace: velero
+spec:
+  scheduleName: authentik-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: music-assistant
+  namespace: velero
+spec:
+  scheduleName: music-assistant-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: immich
+  namespace: velero
+spec:
+  scheduleName: immich-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: influxdb
+  namespace: velero
+spec:
+  scheduleName: influxdb-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: home-assistant
+  namespace: velero
+spec:
+  scheduleName: home-assistant-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: apps
+  namespace: velero
+spec:
+  scheduleName: apps-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: trmnl
+  namespace: velero
+spec:
+  scheduleName: trmnl-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: wordpress
+  namespace: velero
+spec:
+  scheduleName: wordpress-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: navidrome
+  namespace: velero
+spec:
+  scheduleName: navidrome-daily
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: matrix
+  namespace: velero
+spec:
+  scheduleName: matrix-daily
@@ -1,34 +1,30 @@
-# First machine of the throwaway cluster on Hetzner.
+# Hetzner Cloud stand-in for k8s-1. Same link names, addresses and
+# annotations as k8s-1.yaml; the links are WireGuard over the hcloudtest
+# underlay network instead of VLANs, because Hetzner networks are layer 3.
+# Private keys and peers are generated per run by bootstrap.sh hcloudtest.
 #
-# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
-# order bootstrap.sh attaches them.
+# enp1s0 is the public NIC, enp7s0 the underlay. The OSD takes a partition
+# on the local system disk; EPHEMERAL is capped to leave it room.
 machine:
  install:
    disk: /dev/sda
  certSANs:
    - hcloud-1
+  sysctls:
+    net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
+    net.ipv6.conf.ens19.disable_ipv6: "1"
  nodeLabels:
-    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
-    # rather than MAC: the address follows from the order bootstrap.sh attaches
-    # the networks, while the MAC is assigned by Hetzner at attach time and is
-    # not known when this file is written.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+  nodeAnnotations:
+    net.renner.uno/pod0-v6: fd02:48::11
+    net.renner.uno/pod0-v4: 10.2.48.11
 ---
-# In the generated schema rather than machine.network.hostname: naming the host
-# in both is rejected at apply time. `auto: off` switches off the generated
-# `auto: stable`, which is mutually exclusive with a static name.
 apiVersion: v1alpha1
 kind: HostnameConfig
 auto: off
 hostname: hcloud-1
 ---
-# Every interface is stated, the public one included. Talos stops running its
-# default DHCP operators for the whole machine as soon as one new-style network
-# document exists, so an omitted interface gets no address.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp1s0
@@ -37,6 +33,8 @@ up: true
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: enp1s0
+# Below mgmt0: node egress leaves through hcloud-gw, as through the UDM.
+routeMetric: 2048
 ---
 apiVersion: v1alpha1
 kind: LinkConfig
@@ -47,23 +45,73 @@ apiVersion: v1alpha1
 kind: DHCPv4Config
 name: enp7s0
 ---
-# Ceph cluster network, renamed to the role name by the udev rule in the
-# machine section. The node holds no address here; the link exists for the OSD
-# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
 apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
+kind: WireguardConfig
+name: mgmt0
+listenPort: 51840
+mtu: 1370
 up: true
+addresses:
+  - address: 10.2.40.11/24
+  - address: fd40::11/64
+  - address: fd02:40::11/64
+routes:
+  - destination: fd00::1/128
+  - gateway: 10.2.40.1
+  - gateway: fd40::1
+---
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+apiVersion: v1alpha1
+kind: DummyLinkConfig
+name: ens19
+up: true
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: pod0
+listenPort: 51848
+mtu: 1370
+up: true
+addresses:
+  - address: 10.2.48.11/24
+  - address: fd02:48::11/64
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: node0
+listenPort: 51846
+mtu: 1370
+up: true
+addresses:
+  - address: fd02:46::11/64
+  - address: 10.2.46.11/24
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: ingress0
+listenPort: 51847
+mtu: 1370
+up: true
+addresses:
+  - address: 10.2.47.11/24
+  - address: fd02:47::11/64
+---
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  maxSize: 60GiB
+  grow: false
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
 name: osd-data
 provisioning:
  diskSelector:
-    # Two disks: the system disk and the attached volume. Matching on transport
-    # as the qemu patches do selects nothing here — a Hetzner volume arrives
-    # over virtio-scsi, and a selector matching zero disks is skipped silently.
-    match: '!system_disk'
-  # bootstrap.sh reads this size and creates the volume to match.
-  minSize: 50GiB
+    match: system_disk
+  minSize: 345GiB
  grow: true
@@ -1,15 +1,24 @@
-# Second machine. See hcloud-1.yaml.
+# Hetzner Cloud stand-in for k8s-2. Same link names, addresses and
+# annotations as k8s-2.yaml; the links are WireGuard over the hcloudtest
+# underlay network instead of VLANs, because Hetzner networks are layer 3.
+# Private keys and peers are generated per run by bootstrap.sh hcloudtest.
+#
+# enp1s0 is the public NIC, enp7s0 the underlay. The OSD takes a partition
+# on the local system disk; EPHEMERAL is capped to leave it room.
 machine:
  install:
    disk: /dev/sda
  certSANs:
    - hcloud-2
+  sysctls:
+    net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
+    net.ipv6.conf.ens19.disable_ipv6: "1"
  nodeLabels:
    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+  nodeAnnotations:
+    net.renner.uno/pod0-v6: fd02:48::12
+    net.renner.uno/pod0-v4: 10.2.48.12
 ---
 apiVersion: v1alpha1
 kind: HostnameConfig
@@ -24,6 +33,8 @@ up: true
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: enp1s0
+# Below mgmt0: node egress leaves through hcloud-gw, as through the UDM.
+routeMetric: 2048
 ---
 apiVersion: v1alpha1
 kind: LinkConfig
@@ -34,17 +45,73 @@ apiVersion: v1alpha1
 kind: DHCPv4Config
 name: enp7s0
 ---
-# See hcloud-1.yaml.
 apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
+kind: WireguardConfig
+name: mgmt0
+listenPort: 51840
+mtu: 1370
 up: true
+addresses:
+  - address: 10.2.40.12/24
+  - address: fd40::12/64
+  - address: fd02:40::12/64
+routes:
+  - destination: fd00::1/128
+  - gateway: 10.2.40.1
+  - gateway: fd40::1
+---
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+apiVersion: v1alpha1
+kind: DummyLinkConfig
+name: ens19
+up: true
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: pod0
+listenPort: 51848
+mtu: 1370
+up: true
+addresses:
+  - address: 10.2.48.12/24
+  - address: fd02:48::12/64
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: node0
+listenPort: 51846
+mtu: 1370
+up: true
+addresses:
+  - address: fd02:46::12/64
+  - address: 10.2.46.12/24
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: ingress0
+listenPort: 51847
+mtu: 1370
+up: true
+addresses:
+  - address: 10.2.47.12/24
+  - address: fd02:47::12/64
+---
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  maxSize: 60GiB
+  grow: false
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
 name: osd-data
 provisioning:
  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
+    match: system_disk
+  minSize: 345GiB
  grow: true
@@ -1,15 +1,24 @@
-# Third machine. See hcloud-1.yaml.
+# Hetzner Cloud stand-in for k8s-3. Same link names, addresses and
+# annotations as k8s-3.yaml; the links are WireGuard over the hcloudtest
+# underlay network instead of VLANs, because Hetzner networks are layer 3.
+# Private keys and peers are generated per run by bootstrap.sh hcloudtest.
+#
+# enp1s0 is the public NIC, enp7s0 the underlay. The OSD takes a partition
+# on the local system disk; EPHEMERAL is capped to leave it room.
 machine:
  install:
    disk: /dev/sda
  certSANs:
    - hcloud-3
+  sysctls:
+    net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
+    net.ipv6.conf.ens19.disable_ipv6: "1"
  nodeLabels:
    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+  nodeAnnotations:
+    net.renner.uno/pod0-v6: fd02:48::13
+    net.renner.uno/pod0-v4: 10.2.48.13
 ---
 apiVersion: v1alpha1
 kind: HostnameConfig
@@ -24,6 +33,8 @@ up: true
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: enp1s0
+# Below mgmt0: node egress leaves through hcloud-gw, as through the UDM.
+routeMetric: 2048
 ---
 apiVersion: v1alpha1
 kind: LinkConfig
@@ -34,17 +45,73 @@ apiVersion: v1alpha1
 kind: DHCPv4Config
 name: enp7s0
 ---
-# See hcloud-1.yaml.
 apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
+kind: WireguardConfig
+name: mgmt0
+listenPort: 51840
+mtu: 1370
 up: true
+addresses:
+  - address: 10.2.40.13/24
+  - address: fd40::13/64
+  - address: fd02:40::13/64
+routes:
+  - destination: fd00::1/128
+  - gateway: 10.2.40.1
+  - gateway: fd40::1
+---
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+apiVersion: v1alpha1
+kind: DummyLinkConfig
+name: ens19
+up: true
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: pod0
+listenPort: 51848
+mtu: 1370
+up: true
+addresses:
+  - address: 10.2.48.13/24
+  - address: fd02:48::13/64
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: node0
+listenPort: 51846
+mtu: 1370
+up: true
+addresses:
+  - address: fd02:46::13/64
+  - address: 10.2.46.13/24
+---
+apiVersion: v1alpha1
+kind: WireguardConfig
+name: ingress0
+listenPort: 51847
+mtu: 1370
+up: true
+addresses:
+  - address: 10.2.47.13/24
+  - address: fd02:47::13/64
+---
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  maxSize: 60GiB
+  grow: false
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
 name: osd-data
 provisioning:
  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
+    match: system_disk
+  minSize: 345GiB
  grow: true
@@ -1,25 +0,0 @@
-# Site patch for the throwaway cluster on Hetzner Cloud.
-#
-# This site exists for one thing the other two cannot test: whether a node that
-# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
-# answers, so containerd never reaches the second endpoint in
-# machine.registries.mirrors and the fallback stays unproven.
-#
-# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
-# frames from it do not cross between servers — measured: same-node ARP
-# resolves, cross-node does not. OSD replication therefore does not run here.
-#
-# What belongs in this file is what the cluster is. Interfaces, disks and names
-# are machine properties and live in the node patches.
-machine:
-  kubelet:
-    nodeIP:
-      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
-      # kubelet can pick the public NIC.
-      validSubnets:
-        - 10.90.0.0/16
-cluster:
-  # Three control planes that also carry workloads, which is the production
-  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
-  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
-  allowSchedulingOnControlPlanes: true
@@ -10,12 +10,15 @@ machine:
    # The node IP lives on node0, VLAN 46: the closed node↔node network (#576).
    # Both families are listed so the Node object carries an IPv6 InternalIP at
    # all; without one Cilium detects the node's IPv6 itself and picks a SLAAC
-    # address from the dynamic provider prefix (#789). Talos picks one address
-    # per family, IPv4 first, whatever the order here.
+    # address from the dynamic provider prefix (#789).
+    #
+    # Order is load-bearing: Talos keeps this order, the first entry is the
+    # primary node IP, and kube-apiserver requires its family to match the
+    # first service subnet (IPv4, cluster.yaml).
    nodeIP:
      validSubnets:
-        - fd02:46::/64
        - 10.2.46.0/24
+        - fd02:46::/64
  # The UDM on the ingress segment. Published here rather than as a route
  # because bird owns table 47 and originates the default from it; this is the
  # one place the address is stated.
Author	SHA1	Message	Date
claude-botandClaude Opus 5.5	b68ae66b68	fix(hcloud): test nodes take the resolver list from site-homelab.yaml site-homelab.yaml now carries a ResolverConfig (k8s-gateway, then the UDM). The hcloud-N patches are applied after it and each repeated a ResolverConfig with the UDM alone, which would drop k8s-gateway on the test nodes. Removed; the test nodes now resolve like production. hcloud_gw_mgmt_hostroutes read the UDM address from that removed document. It now reads it from site-homelab.yaml, skipping the k8s-gateway ClusterIP taken from its HelmRelease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:11:18 +02:00
claude-botandClaude Opus 5.5	088368fb57	fix(velero): restore Gitea first, the rest after it Gitea is the registry for the in-house images. With one shared restore gate it waited for every volume, Immich's included, while everything pulling from it looped on image pull errors. restore-gitea restores only Gitea and the gitea Kustomization depends on it alone; the other Restores depend on it, so Velero moves Gitea's data first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	271ea43bac	fix(restore-test): the Cloudflare token is synced like in production Withholding it kept the ClusterIssuers, and with them infrastructure-crds, from ever becoming Ready, so nothing after it ran. external-dns writes the same records as production (force-default-targets to the tunnel, upsert-only, same owner id); the tunnel itself does not run in the cluster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	b5b04f995e	fix(hcloud): no GitHub address filter on the gateway's ssh egress github.com resolves to addresses outside the published git ranges (20.250.119.67), so the Flux source timed out. Port 22 is open to all, like 443. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	b184d82fb6	fix(rook): install the ceph-csi-drivers chart Rook requires Since Rook 1.20 the CSI Driver and OperatorConfig resources come from the ceph-csi-drivers chart (rook Documentation/Helm-Charts/csi-drivers-chart.md: "Rook requires the installation of the Ceph-CSI Driver chart"). The chart was never installed here: the live cluster's Drivers are the ones the operator migrated once during the 1.19->1.20 upgrade (2026-05-09), and PR #12 only rendered the ServiceAccounts and RBAC. A fresh cluster therefore has no CSI driver and provisions no volume: found by the Hetzner restore test, where every PVC stayed Pending ("Waiting for a volume to be created either by the external provisioner 'rook-ceph.rbd.csi.ceph.com'"). Values: Rook's recommended values for 1.21.0 plus the live shape: 2 controller replicas with Recreate, the workload-class=pinned toleration on the node plugins, a snapshotter for RBD (Velero's CSI snapshots of ceph-block) and group snapshots for CephFS. The chart's ServiceAccounts are named per driver (rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa, ...), so they do not collide with the static ones; the static ServiceAccounts and RBAC stay until the live CSI pods run on the chart's, and go in a second change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	50456cfa11	fix(restore-test): branch source for every Kustomization, Gitea egress - Every Kustomization of the tree reads the restore-test GitRepository, so FLUX_BRANCH reaches the child paths; before, only the cluster definitions came from the branch and every path from master. - The restored Gitea gets no egress beyond the cluster (3i). Its push mirror pushes with --mirror --force (services/mirror/mirror_push.go, 1.27.3) and would rewind the GitHub mirror to the backup's refs, deleting newer branches, including the one under test. - egress-check: no GitHub HTTPS block any more; asserts a chart release download (302) and that a non-HTTPS port (587) is closed. - Small hardening: snapshot listing failures fail instead of starting a build, builder key dir removed on failure, gateway address check without unbound-variable aborts, yq guard in restore_test_repo, wait-apps does not load access-staging/ for the restore test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	384c398589	feat(restore-test): reconcile FLUX_BRANCH, so a branch is tested before merge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	57a03e81bc	feat(velero): the cluster restores its app volumes itself One Velero Restore per Schedule (kubernetes/restore/restores.yaml), applied by Flux, restoring the schedule's newest completed backup. Every app Kustomization with backed-up volumes depends on the new `restore` Kustomization, so on a fresh cluster no app starts on an empty volume. Restores the capability `952e6150` removed without an owner decision. Ordering and failure handling, from Velero release-1.17 source: - A Restore whose schedule has no synced backup yet goes FailedValidation for good (restore_controller.go). `restore` therefore depends on `velero-backups-synced`, which is Ready once the BackupStorageLocation has completed a sync (status.lastSyncedTime). - After that sync, FailedValidation with "No backups found for schedule" means there is nothing to restore and counts as done; any other FailedValidation, Failed or PartiallyFailed fails the Kustomization. - A claim that already exists is skipped (csi pvc_action: "PVC already exists"), and the Restore completes with a warning. On the running cluster the Restores touch nothing. - Velero deletes a Restore with the backup it came from (30-day TTL); Flux recreates it, and it again skips the existing claims. - All StorageClasses bind Immediate, so the data mover does not wait for a consuming pod. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	9ae8cfa691	fix(gitea): no pinned volumeName on data-gitea-0 The claim named its production PV, which does not exist on a rebuilt cluster: the claim stayed Pending there, and a restored claim bound to a new volume made the gitea Kustomization fail on the immutable field. volumeName leaves git and the gitea Kustomization ignores /spec/volumeName on this claim. kustomize-controller is the field's only manager on the live claim, so its ignore rule adopts the live value instead of clearing it; size growth still applies. To verify after merge: the gitea Kustomization reconciles Ready and the live claim still names pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	d3e31dfa52	fix(restore-test): drop the GitHub filter, harness review findings - hcloud-gw: no nftables filter on GitHub ranges. It also rejected the Helm chart tarballs served from github.com release downloads. - hcloud-token is no longer part of a bare require_secrets: a rebuild of the real cluster must not depend on the OpenBao it rebuilds. - hcloudtest no longer writes access-staging/, which a later production `bootstrap.sh flux` copies over the operator's kube/talos config. - Earlier runs' egress firewalls are removed only after this run's are attached. - Gateway ssh egress limited to GitHub's git ranges (gh api meta). - No partial VIP route to hcloud-1: nothing in the test reaches the VIP through the gateway, and WireGuard cannot follow its election. - Per-process ssh ControlMaster socket, closed on exit. - hcloud_talos_snapshot runs outside $(...) so fail stops the run, and removes the builder server when it fails. - No SIGPIPE-prone `\| head -1` under pipefail; egress check tolerates a pod that never started and reports it. - The restore test's repository is read from flux-root.yaml only. - The osd-data RawVolumeConfig guard is back. - OpenBao answer wait bounded at 30 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	3b4ffc3c41	fix(restore-test): review findings — Cloudflare, access guard, firewall order, gateway Production exposure: - flux-root.yaml never syncs the cloudflare-api-token ExternalSecrets (cert-manager, apps): external-dns and cert-manager would write to the production zone and order production certificates over open HTTPS - the restore-test flux phase refuses production's default kubeconfig AND talosconfig paths, skips load_staged_access (access-staging may hold an older run's files), and checks that every node is an hcloud-* node before it writes the first secret Harness: - this run's firewall is attached to held servers before earlier runs' firewalls are removed; a rebuilt node was briefly unfirewalled in maintenance mode - Talos image write runs with pipefail; a failed download was snapshotted - the gateway's cloud-init is generated in the main shell (fail stops the run), and its runcmd is one && chain so the ready marker means success - the gateway routes only pod CIDRs and the LB pool over ingress0, not node host addresses - default server type cpx52, the type the OSD partition fits - held servers must carry exactly the fixed underlay address Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	545b6acf9d	fix(restore-test): egress check reports the address it reached and always prints its log Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	e50f088c92	fix(hcloud): gateway mgmt0 addresses, node registration race, egress check - the gateway's mgmt0 address list read every mgmt0 route destination; the default routes have none and yielded 'null', which aborted the link setup (fd40::1 and fd02:40::1 were never added) - the cni-confdir pods are created only once every node is registered: a pod bound to a node not yet known is garbage collected and its wait hung - the remote cloud-init wait is bounded at 600s - restore-test/egress-check.yaml asserts the containment from a pod as the last step of hcloudtest: GitHub web/API blocked, Pages, ghcr.io, object storage and GitHub ssh reachable, renner.uno unresolved, egress through hcloud-gw Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	51e02a5aed	fix(bootstrap): re-apply node configs addressed by InternalIP Once Cilium runs, apid hangs on requests whose target (-n) is the node's public IP while the TLS session itself completes; the same request with -n set to the InternalIP through the public endpoint answers at once. The re-apply also gets a 300s bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	4e191f1732	feat(restore-test): contain the test cluster at its gateway - flux-root.yaml: suspend the 13 secret-rotator CronJobs (label app.kubernetes.io/component=secret-rotator); they roll production's tokens at Cloudflare, Authentik, Claude and others - hcloud-gw dnsmasq answers renner.uno locally only; the cluster's k8s-gateway serves its own names, and the public zone's wildcard would lead to production - hcloud-gw nftables rejects forwarded HTTPS to GitHub's web and API ranges (gh api meta), so a restored Gitea push-mirror or bot cannot write to GitHub; Packages and the Fastly /22 (Pages, ghcr blobs) stay reachable - nodes route their default through mgmt0 to hcloud-gw, as production routes through the UDM; replies from the public address use table 100. Before bird fills table 47, pods otherwise left through the public NIC and bypassed the gateway (github.com answered 200 from a pod) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	81623f92d8	refactor(hcloud): egress firewall by port only The Hetzner firewall gets coarse rules: HTTPS, NTP and DNS to the gateway's upstreams for gateway and nodes, plus ssh for the gateway (Flux source). No resolved hostnames, no address lists, no ssh in this step. hcloud-gw's DNS upstreams are stated once and feed both dnsmasq and the firewall. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	4fd96da0b9	fix(hcloud): ControlMaster socket under XDG_RUNTIME_DIR The state-dir path plus %C exceeded the 108-byte Unix socket limit and every gateway command failed with 'too long for Unix domain socket'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	faeeda1e4a	fix(hcloud): no ssh polling of a booting gateway The UDM's IPS blocks the workstation after a burst of ssh connection attempts (wiki: pve-ssh-port22-filtered-udm-ips-block). Wait 90s for the gateway to boot, then at most 4 attempts 60s apart; the cloud-init wait itself runs on the gateway over the shared ControlMaster connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	cf40201e29	fix(hcloud): one multiplexed ssh connection to hcloud-gw per run Hetzner filtered port 22 to hcloud-gw from the workstation after the egress step opened one connection per resolved hostname (~20) on top of 10-second cloud-init polling: 0 pps arrived at the gateway while the Talos API through the same firewall kept answering. All gateway commands now share a ControlMaster connection, the hostname lookups run as one remote command, and the cloud-init wait runs on the gateway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	8d42d89caa	fix(hcloud): fit cpx52 local disks: EPHEMERAL 60GiB, OSD at least 345GiB The project's shared-core quota admits 3x cpx52 (12 vCPU, 24 GB, 480 GB) plus the gateway, not 3x cpx62. Restored data measured from the live kubelets: ~880 GB raw, ~293 GB per node; 345GiB keeps it below 85% nearfull. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	92c7ee7314	feat(hcloud): OSDs on the local system disk, no Hetzner volumes The project's volume quota is 1024 GB, short of three OSD volumes sized for the restored data. cpx62 carries 640 GB of local NVMe: EPHEMERAL is capped at 100GiB and the OSD takes a raw partition of at least 400GiB on the system disk. The harness creates Hetzner volumes only for raw volumes off the system disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	ac1e3b8467	fix(hcloud): 400GiB OSD volumes, sized for the restored data only Restored: 323 GB bulk libraries on the 1.5x EC pool, ~92 GB other Velero PVCs at 3x, plus the CNPG databases: ~1 TB raw, ~320 GB per node. Prometheus, Loki and llama-server's models are not restored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	6f31df224d	feat(hcloud): size the OSD volumes for the restored data (800GiB per node) Production Ceph holds 1.6 TiB raw (ceph df, 2026-10-15); 3x800GiB keeps the restored cluster below the 85% nearfull ratio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	4aa63e0adc	feat(restore-test): close the test cluster's egress with the Hetzner firewall hcloud-gw carries all pod traffic and gets an allowlist resolved on the gateway: object storage and the Helm chart sources (read from the tree's HelmRepository/OCIRepository objects) on 443, GitHub on 22 only (the Flux source; without 443 a restored Gitea push-mirror cannot reach GitHub), its DNS upstreams and NTP. The nodes get HTTPS and NTP for image pulls from CDN-backed registries, plus DNS for maintenance mode; only host processes use that path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	24f7988909	feat(restore-test): the test cluster reads the production backups, never writes them The restored OpenBao hands the test cluster production's object-storage credentials. flux-root.yaml now patches every backup writer off, inside the same single patch block that removes the Ceph cluster network: - Velero BSL accessMode ReadOnly: blocks backups, deletion, GC and kopia repository maintenance; restores and backup sync still work (velero release-1.17 backup/gc/deletion/repository controllers) - CNPG Clusters: isWALArchiver false; the plugin stays loaded for recovery - CNPG ObjectStores: retentionPolicy removed; the plugin sidecar enforces it on the primary whether or not it archives - CNPG ScheduledBackups suspended; the suspend check precedes the immediate backup in the controller - MariaDB Backup schedule suspended Verified with flux build on infrastructure, authentik, woodpecker, wordpress and grafana-db. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	aa6177e0c0	feat(restore-test): one file holds the off-site test's deviation from production kubernetes/restore-test/flux-root.yaml is the test cluster's Flux root. It reconciles ./kubernetes/clusters/homelab unchanged and lists every difference as a patch: 1. the flux-system GitRepository reads the GitHub mirror through a per-run, read-only deploy key 2. the tree's own flux-system Kustomization is suspended 3. the CephCluster loses spec.network: Hetzner Cloud networks carry no layer 2, so the ceph-san attachment cannot reach other nodes and Ceph replicates over the pod network Verified with flux build: the rendered tree differs from a production render in exactly those three objects. bootstrap.sh flux gains FLUX_SOURCE=restore-test (refuses the operator's production kubeconfig); hcloudclean revokes the deploy keys. The Hetzner site drops its storage network and the ceph-san NIC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:30 +02:00
claude-botandClaude Opus 5.5	ce9bce23e2	feat(bootstrap): Hetzner rehearsal site rebuilt as the production network in WireGuard hcloudtest now provisions the production network shape instead of a flat one: each production segment (mgmt0, node0, pod0, ingress0) is a WireGuard mesh over a private underlay with the production link names and addresses, ens19 is a dummy link, and hcloud-gw stands in for the UDM (10.2.40.1 DNS, 10.2.47.1 BGP peer and egress NAT, fd00::1, fd02:47::1). Nodes take site-homelab.yaml unchanged; site-hcloud.yaml is gone. Runs in the Hetzner project homelab-backup-verify (token from OpenBao). The Talos snapshot is built automatically when missing. Cold-start fixes that a real rebuild from git needs as well: - create Cilium's cni.confPath (/etc/cni/net.d/multus) before installing Cilium: the kubelet sees /etc/cni read-only on Talos and cannot create it - install Multus right after Cilium: it is the only writer of /etc/cni/net.d (#832), and without it no pod schedules, Flux included Verified on Hetzner 2026-10-15: etcd, API servers, Cilium native routing, cross-node pod traffic v4/v6, DNS via the gateway, pod egress via table 47, 6/6 BGP sessions, LoadBalancer IPs reaching backends on all nodes. Not working: the Ceph SAN host-device attachment (Hetzner networks do not answer ARP between servers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:29 +02:00
claude-botandClaude Opus 5.5	de237a2d44	fix(talos): IPv4 first in nodeIP.validSubnets, as the live nodes run Talos keeps the validSubnets order (siderolabs/net FilterIPs) and the first entry becomes the primary node IP. Git had fd02:46::/64 first since `be643533` while k8s-1 runs with 10.2.46.0/24 first. A cluster built from git gets an IPv6 primary and kube-apiserver exits with 'service IP family "10.96.0.0/12" must match public address family "fd02:46::11"' — reproduced on a Hetzner rebuild rehearsal 2026-10-15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 08:10:29 +02:00