docs: per-site base domain and service CIDR for the restore test (design + plan)

The restore-test site gets its own base domain (rtest.renner.uno), service
CIDR, NetBird identity and ACME server as site values in the Talos site
patches, and the features it lacks (iGPU, IoT VLAN, UniFi gateway) are
suspended there. Production renders unchanged. Approved by the owner
2026-10-18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/superpowers/plans/2026-10-18-site-base-domain.md

@@ -0,0 +1,162 @@
+# Site-specific base domain and service CIDR Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** The Hetzner restore-test site runs production's tree under its own base domain, service CIDR, NetBird identity and ACME server, with the features it lacks suspended, so it can be reached through the NetBird proxy without touching production.
+
+**Architecture:** Each Talos site patch carries one more inline manifest, ConfigMap `flux-system/site-settings` (plus the `flux-system` Namespace), holding the site's values as `SITE_*` variables. Flux reads it through `postBuild.substituteFrom`: the root sync substitutes it into the Kustomization objects (for `spec.suspend`), and every Kustomization whose manifests carry a site value substitutes it into its manifests. Production's values reproduce today's literals, so its rendered output is byte-identical; a render check enforces that per PR.
+
+**Tech Stack:** Flux v2 (kustomize-controller v1.9.5, flux CLI ≥ v2.8 for `flux envsubst --strict`), kustomize via `kubectl kustomize`, Talos inline manifests, bash, yq (both flavours must work), Woodpecker CI.
+
+**Spec:** `docs/superpowers/specs/2026-10-18-site-base-domain-design.md`
+
+## Global Constraints
+
+- Only what must differ, differs. Every `SITE_*` variable exists because of a collision named in the spec.
+- Production (`site-homelab`) renders byte-identical to `origin/master` for every changed path, except the two intended pod-spec additions named in Task 3 (Gitea `GITEA__server__ROOT_URL`, gate-sync `COOKIE_DOMAIN`), each set to the value already in effect.
+- Site values live in the Talos site patches (`kubernetes/talos/patches/site-{homelab,hcloud,qemu}.yaml`). No test-only patch in `restore-test/flux-root.yaml` may introduce a site difference.
+- Never force-push; stacked PRs, each based on the previous; never self-merge.
+- Moving objects between pruned Kustomizations is not part of this plan; if a task finds it needs one, stop and use the #1307 prune handover.
+- Talos changes reach production as incremental `talosctl patch machineconfig` (never the whole repo file), `--dry-run` output filtered so no machine-config secret is printed, k8s-1 first, then k8s-2, k8s-3.
+- Matrix stays hardcoded (`server_name: matrix.renner.uno`); out of scope.
+- Comments state facts, not history. No prose in config files.
+
+Site values (exact):
+
+| Variable | homelab | hcloud | qemu |
+|---|---|---|---|
+| `SITE_DOMAIN` | `renner.uno` | `rtest.renner.uno` | `renner.uno` |
+| `SITE_ACME_SERVER` | `https://acme-v02.api.letsencrypt.org/directory` | `https://acme-staging-v02.api.letsencrypt.org/directory` | `https://acme-v02.api.letsencrypt.org/directory` |
+| `SITE_SERVICE_CIDR` | `10.96.0.0/12` | `10.112.0.0/12` | `10.96.0.0/12` |
+| `SITE_DNS_GATEWAY_IP` | `10.96.0.11` | `10.112.0.11` | `10.96.0.11` |
+| `SITE_NETBIRD_NETWORK` | `k8s public ingress` | `k8s public ingress rtest` | `k8s public ingress` |
+| `SITE_NETBIRD_ROUTER_GROUP` | `k8s-ingress-router` | `k8s-ingress-router-rtest` | `k8s-ingress-router` |
+| `SITE_NETBIRD_INGRESS_KEY` | `infrastructure/netbird-ingress` | `infrastructure/netbird-ingress-rtest` | `infrastructure/netbird-ingress` |
+| `SITE_SUSPEND_IGPU` | `false` | `true` | `true` |
+| `SITE_SUSPEND_IOT_VLAN` | `false` | `true` | `true` |
+| `SITE_SUSPEND_UNIFI` | `false` | `true` | `true` |
+
+`10.112.0.0/12` overlaps nothing in the repo (checked: Ceph SAN `10.99.0.0/29` sits inside `10.96.0.0/12`, Hetzner nodes use `10.90.0.0/16`, NetBird `100.64.0.0/10`).
+
+## Review Focus
+
+1. A manifest in a newly substituting Kustomization contains unrelated `${...}` (shell scripts, Grafana dashboards, proxy configs, CEL) — Flux would blank it silently. Expected: `flux envsubst --strict` in the render check fails on it; the object carries `kustomize.toolkit.fluxcd.io/substitute: disabled` or the `$` is escaped as `$${...}`. Test: Task 1's `test-site-render.sh` case `strict_fails_on_unknown_var`.
+2. A suspended Kustomization is a `dependsOn` target of one that is not suspended on the same site — the dependent waits forever. Expected: nothing unsuspended depends on a suspended one. Test: Task 2 case `no_dependents_of_suspended`.
+3. A served name left hardcoded on the hcloud site (an Ingress host, redirect URI, probe target) — the test publishes or calls a production name. Expected: the hcloud render contains `renner.uno` only as `rtest.renner.uno`, in the allow-list (label/annotation keys, image references, the Git source, email addresses, external hosts, Matrix). Test: Task 3 case `hcloud_has_no_production_names`.
+4. The restore-test root (`restore-test/flux-root.yaml`) lacks the substitution the production root has, so suspends and domains render empty on the test. Expected: both roots substitute `site-settings`. Test: Task 2 case `restore_root_substitutes`.
+5. The test's NetBird peers gain access to production through NetBird's "All" group policies. Expected: verified against the live NetBird policies before run 31 (Task 6 Step 2).
+
+---
+
+### Task 1: Render check for site values
+
+**Files:**
+- Create: `scripts/ci/site-render.sh`
+- Create: `scripts/ci/test-site-render.sh`
+- Create: `scripts/ci/testdata/site-render/` (fixtures)
+- Modify: `.woodpecker/ci-manifests-test.yml` (new step `site-render`)
+
+**Interfaces:**
+- Produces: `scripts/ci/site-render.sh <site> <path>` — prints `kubectl kustomize <path>` piped through `flux envsubst --strict` with the `SITE_*` values read from the `site-settings` inline manifest of `kubernetes/talos/patches/site-<site>.yaml`; exit non-zero on an undefined variable. With `--root`, renders `kubernetes/clusters/homelab` the way the root sync does.
+- Produces: `scripts/ci/site-render.sh --changed <site> <base-ref>` — for every Flux Kustomization path whose object has `postBuild.substituteFrom` naming `site-settings`, diff the render at HEAD against the plain render at `<base-ref>`; exit 1 on a difference, printing it.
+
+- [ ] **Step 1: Write the failing tests** in `scripts/ci/test-site-render.sh` (same style as `scripts/ci/test-yaml-citations.sh`), with fixtures under `scripts/ci/testdata/site-render/`:
+  - `renders_site_value`: fixture kustomization with `host: app.${SITE_DOMAIN}` and a fixture site patch with `SITE_DOMAIN: example.test` → output contains `host: app.example.test`.
+  - `strict_fails_on_unknown_var`: fixture containing `${FOO}` → exit status non-zero, stderr names `FOO`.
+  - `escaped_dollar_survives`: fixture containing `$${FOO}` → output contains `${FOO}`.
+  - `disabled_object_untouched`: object annotated `kustomize.toolkit.fluxcd.io/substitute: disabled` containing `${FOO}` → output keeps `${FOO}`, exit 0.
+- [ ] **Step 2: Run** `bash scripts/ci/test-site-render.sh` — expected FAIL (`site-render.sh` missing).
+- [ ] **Step 3: Implement `site-render.sh`.** Read the ConfigMap from the site patch with the same yq expression bootstrap.sh uses for `cilium-site` (`select(.cluster) | .cluster.inlineManifests[] | select(.name == "site-settings") | .contents`), no `-y`; export its `data` keys; honour the `substitute: disabled` annotation by rendering those objects untouched (split the stream with yq, substitute the rest). Never `2>/dev/null`.
+- [ ] **Step 4: Run** `bash scripts/ci/test-site-render.sh` — expected PASS; `shellcheck -S warning scripts/ci/site-render.sh scripts/ci/test-site-render.sh` clean.
+- [ ] **Step 5: Add CI step** `site-render` to `.woodpecker/ci-manifests-test.yml` running the test script, then `site-render.sh --changed homelab "$CI_COMMIT_TARGET_BRANCH"` on pull requests. Image must carry flux and yq (check `ci/images/ci-k8s`; add flux there if missing, in this task).
+- [ ] **Step 6: Commit** `ci(manifests): render check for per-site values`.
+
+### Task 2: site-settings ConfigMap, root substitution, feature suspension
+
+**Files:**
+- Modify: `kubernetes/talos/patches/site-homelab.yaml`, `site-hcloud.yaml`, `site-qemu.yaml` (new inline manifest `site-settings`: Namespace `flux-system` + ConfigMap with all `SITE_*` keys from the table)
+- Modify: `kubernetes/clusters/homelab/flux-system/kustomization.yaml` (patch adding `postBuild.substituteFrom: [{kind: ConfigMap, name: site-settings}]` to Kustomization `flux-system`; bootstrap keeps this file)
+- Modify: `kubernetes/restore-test/flux-root.yaml` (same `postBuild` on Kustomization `restore-test`)
+- Modify: `kubernetes/clusters/homelab/llama-server.yaml`, `wiki.yaml` (`suspend: ${SITE_SUSPEND_IGPU}`), `home-assistant.yaml` (`matter-server`, `home-assistant-iot`: `${SITE_SUSPEND_IOT_VLAN}`), `music-assistant.yaml` (`${SITE_SUSPEND_IOT_VLAN}`), `infrastructure-services.yaml` (`external-dns-unifi`: `${SITE_SUSPEND_UNIFI}`)
+- Test: `scripts/ci/test-site-render.sh`
+
+**Interfaces:**
+- Consumes: `site-render.sh` (Task 1).
+- Produces: ConfigMap `flux-system/site-settings` with the keys in the table; root substitution available to every Kustomization object in `kubernetes/clusters/homelab`.
+
+- [ ] **Step 1: Write the failing tests:**
+  - `root_suspends_per_site`: `site-render.sh --root hcloud` shows `suspend: true` on the six Kustomizations listed above; `--root homelab` shows `suspend: false` on them and `otbr` unchanged (`true`).
+  - `restore_root_substitutes`: `kubernetes/restore-test/flux-root.yaml`'s `restore-test` Kustomization has `postBuild.substituteFrom` naming `site-settings` (yq query, exit 0).
+  - `no_dependents_of_suspended`: for each site, no Kustomization with `suspend: false` lists a suspended one in `dependsOn`.
+  - `homelab_root_unchanged`: `--root homelab` render equals the plain render at `origin/master` except `suspend: false` on the six objects.
+- [ ] **Step 2: Run tests** — expected FAIL.
+- [ ] **Step 3: Add the inline manifest and the suspend fields** listed under Files. Production values make `suspend: false` explicit; Flux treats it like the absent field.
+- [ ] **Step 4: Run tests** — expected PASS; `site-render.sh --root homelab` exits 0 (no undefined variable anywhere in `kubernetes/clusters/homelab`).
+- [ ] **Step 5: Commit** `feat(flux): site-settings from the Talos site patch; features a site lacks are suspended`.
+- [ ] **Step 6: Production rollout before merge** (PR body): incremental Talos patch adding only the `site-settings` inline manifest, dry-run filtered, k8s-1 then k8s-2, k8s-3; verify `kubectl -n flux-system get cm site-settings -o yaml` equals the homelab column.
+
+### Task 3: Base domain and ACME server per site
+
+**Files (served names only; the spec's inventory lists them, about 120 files):**
+- Modify: every Ingress/TLS host, cert-manager `dnsZones` and `dnsNames`, external-dns `domainFilters` and hostname annotations, k8s-gateway zone (`infrastructure/k8s-gateway/helmrelease.yaml:56,61`), CoreDNS zone (`infrastructure/coredns-config/configmap.yaml:28`), app URLs, authentik blueprint redirect URIs, `authentik_host_browser`, brand domain, LDAP names, monitoring probes, gatus, uptime-monitor — `renner.uno` → `${SITE_DOMAIN}`.
+- Modify: `infrastructure/post-crds/clusterissuer.yaml` `${ACME_SERVER}` → `${SITE_ACME_SERVER}`; `infrastructure/cluster-settings/cluster-settings.yaml` drops `ACME_SERVER`; `clusters/homelab/infrastructure.yaml` `infrastructure-crds` adds `site-settings` to `substituteFrom`.
+- Modify: `apps/gitea/statefulset.yaml` add `GITEA__server__ROOT_URL: https://gitea.${SITE_DOMAIN}`.
+- Modify: `apps/authentik/gate-sync.yaml` add `COOKIE_DOMAIN: ${SITE_DOMAIN}`.
+- Modify: each `kubernetes/clusters/homelab/*.yaml` Kustomization whose path now contains `${SITE_DOMAIN}` — add `postBuild.substituteFrom: [{kind: ConfigMap, name: site-settings}]`.
+- Modify: objects in those paths with unrelated `${...}` (list in the spec's review and `git grep -l '\${[A-Za-z_]'`: home-assistant deployment/proxy.conf/config-sync.sh/zigbee2mqtt scripts, home-assistant-iot shelly proxy conf, otbr, monitoring dashboards and helmrelease-prometheus, multus cni-plugins-installer, openbao snapshot-cronjob, rook toolbox, image-automation lidarr, velero backup-verify) — annotate `kustomize.toolkit.fluxcd.io/substitute: disabled` where the object needs no site value, else escape as `$${...}`.
+- Unchanged by rule: label/annotation keys (`dns.renner.uno/*`, `auth.renner.uno/*`, `net.renner.uno/*`, `cnpg.renner.uno/*`, …), image references, `gotk-sync.yaml` Git URL, email addresses, names of hosts outside the cluster (udm, nas-1/2, pve, jupiter/saturn/neptune, `netbird.renner.uno`, `slzb-*`, `*.iot.renner.uno`, `edge-1`), Matrix.
+
+**Interfaces:**
+- Consumes: `site-settings` keys `SITE_DOMAIN`, `SITE_ACME_SERVER` (Task 2); `site-render.sh` (Task 1).
+
+- [ ] **Step 1: Write the failing test** `hcloud_has_no_production_names` in `test-site-render.sh`: render every substituting path with `hcloud`; every `renner.uno` occurrence must be `rtest.renner.uno` or match the allow-list regex file `scripts/ci/testdata/site-render/allowed-production-names.txt` (one regex per line, each with a comment naming why it stays).
+- [ ] **Step 2: Run** — expected FAIL with the list of leftover names.
+- [ ] **Step 3: Replace** per the Files list, in reviewable commits by area: infrastructure (DNS, cert-manager, ACME), authentik, apps, monitoring.
+- [ ] **Step 4: Run** `test-site-render.sh` — PASS; `site-render.sh --changed homelab origin/master` — exits 0 except the two intended env additions, which it prints; confirm their values equal production's (`gitea.renner.uno`, `renner.uno`).
+- [ ] **Step 5: Commit** per area, `feat(site): <area> reads the base domain from site-settings`.
+
+### Task 4: Service CIDR per site
+
+**Files:**
+- Modify: `kubernetes/talos/patches/cluster.yaml:80-81` (remove `serviceSubnets`), `site-homelab.yaml`, `site-qemu.yaml` (`serviceSubnets: [10.96.0.0/12]`), `site-hcloud.yaml` (`[10.112.0.0/12]`, `ResolverConfig` nameserver `10.112.0.11` at :78)
+- Modify: `infrastructure/k8s-gateway/helmrelease.yaml:104` `clusterIP: ${SITE_DNS_GATEWAY_IP}`; `infrastructure/coredns-config/configmap.yaml:35` `forward . ${SITE_DNS_GATEWAY_IP}`; `infrastructure-inhouse/netbird-service-sync/deployment.yaml:99` `${SITE_SERVICE_CIDR}`; their Kustomizations get `site-settings` in `substituteFrom` if Task 3 did not add it.
+- Modify: `kubernetes/bootstrap.sh` k8s-gateway install (~1459): pass the site's `SITE_DNS_GATEWAY_IP` as the Service clusterIP (read from `site-settings` with the same yq expression as Task 1).
+
+**Interfaces:**
+- Consumes: `SITE_SERVICE_CIDR`, `SITE_DNS_GATEWAY_IP`.
+
+- [ ] **Step 1: Failing tests:** `service_cidr_per_site` — `yq` over the merged Talos patches for each site yields the table's CIDR exactly once; `hcloud` render of k8s-gateway has `clusterIP: 10.112.0.11` and CoreDNS forwards there.
+- [ ] **Step 2: Run** — FAIL.
+- [ ] **Step 3: Implement** per Files.
+- [ ] **Step 4: Run** — PASS; homelab render unchanged; `shellcheck -S warning kubernetes/bootstrap.sh`.
+- [ ] **Step 5: Commit** `feat(site): service CIDR and DNS gateway address per site`.
+- [ ] **Step 6: Production rollout:** none on the nodes — production's `serviceSubnets` value is unchanged, only moved between patch files; the live machine config already holds it.
+
+### Task 5: NetBird identity per site
+
+**Files:**
+- Modify: `infrastructure-inhouse/netbird-service-sync/deployment.yaml` env `NETWORK_NAME: ${SITE_NETBIRD_NETWORK}`, `ROUTER_GROUP_NAME: ${SITE_NETBIRD_ROUTER_GROUP}`.
+- Modify: `infrastructure/netbird-ingress/externalsecret.yaml` `key: ${SITE_NETBIRD_INGRESS_KEY}`; `clusters/homelab/infrastructure-services.yaml` `netbird-ingress` gets `site-settings` substitution.
+- Create: `infrastructure/openbao/rotation/netbird-ingress-rtest.yaml` — a second CronJob like `netbird-ingress.yaml`, `BAO_PATH: homelab/infrastructure/netbird-ingress-rtest`, `NB_AUTO_GROUPS: <id of group k8s-ingress-router-rtest>`; add it to `infrastructure/openbao/rotation/kustomization.yaml`; extend the `rotator-netbird` OpenBao policy (`infrastructure/openbao/policies/`) to write that path.
+
+**Interfaces:**
+- Consumes: `SITE_NETBIRD_*` (Task 2).
+- Produces: OpenBao key `homelab/infrastructure/netbird-ingress-rtest` holding a setup key whose only auto-group is `k8s-ingress-router-rtest`.
+
+- [ ] **Step 1: Owner step (blocking):** the owner creates NetBird group `k8s-ingress-router-rtest` with no policies granting it access to production resources and gives its ID. Do not create it with credentials read from OpenBao.
+- [ ] **Step 2: Failing tests:** hcloud render of netbird-service-sync has `NETWORK_NAME=k8s public ingress rtest`, `ROUTER_GROUP_NAME=k8s-ingress-router-rtest`; the netbird-ingress ExternalSecret extracts `infrastructure/netbird-ingress-rtest`; homelab render unchanged.
+- [ ] **Step 3: Implement** per Files.
+- [ ] **Step 4: Run tests** — PASS.
+- [ ] **Step 5: Commit** `feat(site): NetBird network, router group and setup key per site`.
+- [ ] **Step 6: Production rollout after merge:** create one Job from the new CronJob (`kubectl -n openbao create job --from=cronjob/netbird-secret-rotate-rtest …`), confirm the OpenBao key exists (metadata only, never the value), so the next raft snapshot carries it into the test.
+
+### Task 6: Restore test with the new site (run 30, then run 31)
+
+**Files:**
+- Modify: `kubernetes/restore-test/flux-root.yaml` — run 31 only: remove patches 3f and 3g; renumber.
+
+- [ ] **Step 0: Before run 30:** list every object carrying the label `external-dns/cloudflare=true` (the Cloudflare instance's `labelFilter`) and the names it would publish on hcloud. With `--default-targets` set to a `*.cfargotunnel.com` name and `--force-default-targets` (`apps/external-dns/helmrelease.yaml`), each such record would shadow the public wildcard with an unreachable target; if any exists, stop and bring it to the owner.
+- [ ] **Step 1: Run 30** with patches 3f/3g still in place, hands-off, stall detector armed. Pass criteria: no undefined-variable errors; the six features show suspended; every other Kustomization Ready; all names on the test are `*.rtest.renner.uno`; certificates come from LE staging; production unaffected (production probes and `flux get ks` on production stay green); stale production URLs in restored data fail from the test (a pod on the test reaching `https://ci.renner.uno` and `https://auth.renner.uno` times out at the egress firewall). Record results in the wiki.
+- [ ] **Step 2: Before run 31:** owner and builder read the live NetBird policies for group "All" and `k8s-ingress-router-rtest`; the test's peers must reach nothing of production. Check whether the NetBird proxy accepts services under `rtest.renner.uno` without a separately registered custom domain; if registration is needed, it is the owner's step.
+- [ ] **Step 3: Remove patches 3f/3g**, commit `fix(restore-test): NetBird sync and routing peers run on the test site`, PR.
+- [ ] **Step 4: Run 31** hands-off. Production checks every 5 min (NetBird service list for production hosts unchanged, `netbird-service-sync` log in production, public probes). Stop and tear down on the first production deviation. Pass criteria additionally: the owner opens `https://<app>.rtest.renner.uno` from the internet and logs in with the password, and an app behind the passkey step-up (`passkey-stepup.yaml`: e.g. Gitea) lets them register a new passkey on the test host after password login.

docs/superpowers/specs/2026-10-18-site-base-domain-design.md

@@ -0,0 +1,227 @@
+# Site-specific base domain and address ranges for the restore test
+
+Date: 2026-10-18. Status: design, awaiting the owner's approval.
+
+## Intent
+
+The Hetzner restore test (`bootstrap.sh hcloudtest` + `FLUX_SOURCE=restore-test bootstrap.sh flux`)
+boots production's configuration on throwaway machines and restores production's data into it.
+Today it has to run with parts of production switched off (`kubernetes/restore-test/flux-root.yaml`
+patches 3f and 3g, external-dns left unresolved), because it uses production's names and would
+overwrite production's NetBird services, routing and DNS. In run 29 production ingress broke during
+the test and the cause was never found.
+
+The fix is to give the test cluster its own identity where, and only where, it collides with
+production: a subdomain as its base domain and its own address ranges. Then the test can run with
+everything production runs, including the internet path through the NetBird proxy, and the owner can
+open the restored services from the internet to check that they actually work.
+
+## Goals
+
+- The base domain is a site fact. Production (`site-homelab`) keeps `renner.uno`. The restore-test
+  site (`site-hcloud`) uses a subdomain of `renner.uno` (working name `rtest.renner.uno`; the builder
+  confirms the final label with the owner only if it matters).
+- The address range that collides is a site fact: the service CIDR, and every fixed address derived
+  from it. Pod CIDR and LoadBalancer pool are not routed into NetBird for the test site (see 3), so
+  they collide nowhere and stay identical.
+- The NetBird identity of the cluster is per site, so production and test live side by side in the
+  same NetBird account.
+- The test cluster is reachable from the internet through the same NetBird proxy as production, under
+  `*.<test domain>` names, and works like production there.
+- With these in place, the restore-test patches that exist only because of name or range collisions
+  are removed (3f netbird-service-sync at zero replicas, 3g netbird-ingress on no node), and the
+  Cloudflare external-dns runs on the test site. external-dns-unifi does not, and neither do the
+  features the Hetzner site lacks: they are suspended there (see Decided).
+
+## Non-goals
+
+- Making production serve several domains.
+- Matrix (Synapse, MAS, Element): out of scope (owner, 2026-10-18). Synapse refuses to start if
+  `server_name` changes (upstream `synapse/storage/databases/main/__init__.py`), so it stays
+  `matrix.renner.uno`; Matrix may be broken or unreachable on the test site.
+- Passkeys on the test site: authentik's WebAuthn RP ID is the request host
+  (`authentik/stages/authenticator_webauthn/utils.py:6-11`, version 2026.8.3, not configurable), so
+  passkeys registered on `auth.renner.uno` do not work on the test host. The owner logs in with
+  the password; passkeys are not a design criterion (owner, 2026-10-18).
+- Rewriting production URLs inside restored data (see Constraints, "Stale production URLs").
+
+## Anti-goals (a failure even if it technically works)
+
+- Any difference between the sites that is not needed. "Only things that must differ should differ"
+  (owner). Each per-site value must be justified by a collision.
+- A harness or script that changes the test nodes or network to make the test pass. The test must
+  boot production's configuration; per-site facts live in the site and node files
+  (`site-homelab` describes the local cluster, `site-hcloud` the Hetzner cluster, node specifics in
+  their own files — owner's rule).
+- The test cluster writing anything to production: production's NetBird services, routes, peers or
+  groups; production's DNS names; production's Woodpecker, Gitea or other services.
+- The cluster depending on external DNS for its own services (owner's design rule).
+- Any change in production's behaviour. For production, the change is a pure move of values into
+  the site files; rendered manifests stay identical.
+
+## What has to differ per site, and why
+
+### 1. Base domain
+
+Collision: the test's Ingress hosts, NetBird proxy services (keyed by domain) and DNS records would be
+production's.
+
+Where the domain is written (inventory at origin/master, 2026-10-18, about 160 files contain
+`renner.uno`; only the served names below change per site):
+
+- Ingress and TLS hosts: about 34 files under `kubernetes/apps` and `kubernetes/infrastructure`
+  (e.g. `apps/gitea/ingress.yaml:19,29`), including nested names `*.home.renner.uno`.
+- cert-manager: `infrastructure/post-crds/clusterissuer.yaml` (`dnsZones: renner.uno` at 19, 39, 66;
+  the ACME email at 8, 28, 55 is an address, not a served name); Certificates with `dnsNames`
+  (`apps/authentik/ldap-outpost-cert.yaml:28`, `apps/home-assistant/mosquitto-certificate.yaml:12`,
+  `apps/monitoring/helmrelease-alloy-syslog.yaml:12`).
+- DNS: both external-dns instances (`apps/external-dns/helmrelease.yaml:51` Cloudflare;
+  `infrastructure/external-dns-unifi/helmrelease.yaml:84`), the `external-dns.kubernetes.io/hostname`
+  annotations (6 files), k8s-gateway zone (`infrastructure/k8s-gateway/helmrelease.yaml:56,61`),
+  CoreDNS zone (`infrastructure/coredns-config/configmap.yaml:28`).
+- Application config with URLs: Gitea, Woodpecker, Immich, Paperless, WordPress, TRMNL, gitea-mcp,
+  review-bot, wiki, renovate, Grafana and the monitoring probes, gatus, uptime-monitor.
+- authentik blueprints (`apps/authentik/blueprints-configmap.yaml`): OAuth redirect URIs (~813-951),
+  LDAP name and `tls_server_name`, `authentik_host_browser`, brand domain (1170).
+- authentik gate-sync: `AUTHENTIK_HOST_BROWSER` (`apps/authentik/gate-sync.yaml:127`) and
+  `COOKIE_DOMAIN`, which is unset and defaults to `renner.uno` in the controller (`controller.py:57`);
+  it must be set explicitly.
+- Gitea `ROOT_URL` lives in `app.ini` on the PVC (restored data), not in git; git must set it
+  (`GITEA__server__ROOT_URL`) so the test serves its own URL.
+
+Stays the same on every site (not served names): label and annotation keys such as
+`dns.renner.uno/public`, `auth.renner.uno/*`, `net.renner.uno/*`, `cnpg.renner.uno/*`; image
+references and the registry mirror (`gitea.renner.uno` as a pull source); Flux's Git source;
+email addresses; comments and docs. Names of things outside the cluster (UDM, NAS, PVE hosts,
+IoT devices, `netbird.renner.uno`) stay too.
+
+### 2. Service CIDR
+
+Collision: the NetBird proxy reaches a cluster through a route to its service CIDR and connects to
+Traefik's ClusterIP inside it (`ci/images/netbird-service-sync/controller.py`, `SERVICE_CIDR`,
+`service_body()`). Two clusters with the same CIDR give the proxy two routes to the same range; it
+cannot tell the clusters apart.
+
+Currently shared, to become site facts:
+
+- `serviceSubnets: 10.96.0.0/12` in `kubernetes/talos/patches/cluster.yaml:80-81`.
+- Fixed service addresses: k8s-gateway `10.96.0.11` (`infrastructure/k8s-gateway/helmrelease.yaml:104`),
+  CoreDNS forward to it (`infrastructure/coredns-config/configmap.yaml:35`), the Talos
+  `ResolverConfig` in the site patches, kube-dns `10.96.0.10`.
+- `netbird-service-sync` `SERVICE_CIDR` env in its Deployment.
+
+Stay identical (no collision, because the test site routes nothing else into NetBird): pod CIDR
+`10.244.0.0/16`, LoadBalancer pool `10.2.99.0/24` / `fd99::/64`
+(`infrastructure/post-crds/cilium-bgp.yaml:22-25`) and the VIPs in it (Traefik `10.2.99.86`, Home
+Assistant `10.2.99.87`). The builder confirms nothing else routes these between the sites; if
+something does, that is a collision and the range becomes a site fact too.
+
+Production keeps its current value. Its service CIDR cannot change on a running cluster anyway.
+
+### 3. NetBird identity
+
+Collision (verified in `ci/images/netbird-service-sync/controller.py`): the controller finds its
+network by name (`NETWORK_NAME`, default `k8s public ingress`, line 44) and its resource by the
+service CIDR; it treats every TLS service targeting that resource as its own and deletes those no
+local Ingress carries (`is_managed` line 212, delete line 257). A second cluster with the same name
+would delete production's services. `netbird-ingress` peers enrol with a setup key whose auto-group
+is `k8s-ingress-router`, the router group of production's network; test peers would become routers
+for production's traffic.
+
+Per site: `NETWORK_NAME`, `ROUTER_GROUP_NAME` (and the policy name derived from the network name), the
+`netbird-ingress` setup key and its auto-group.
+
+Decided (owner, 2026-10-18): the test site gets its own setup key that only allows access to where it
+is needed. The secret rotator creates it (production's key is minted by
+`infrastructure/openbao/rotation/netbird-ingress.yaml`, `NB_AUTO_GROUPS`). The test's peers get no
+production group, no production policy, no mesh routes and no mesh DNS: only what the proxy path to
+the test's own services needs. Production's routes and nameserver groups for `k8s-ingress-router` are
+set up by hand in NetBird, not in the repo; the test's peers must not be in that group. Peers join
+NetBird's "All" group automatically; the builder checks what "All" grants in production and makes
+sure the test's peers gain no access to production through it.
+
+## Constraints
+
+- Production config is the spec. Per-site values are expressed in the site files (`site-homelab.yaml`,
+  `site-hcloud.yaml`, `site-qemu.yaml`), the way `cilium-site` and `k8s-gateway-site` already are
+  (inline ConfigMaps read via `valuesFrom`, Talos-created). The builder chooses the mechanism by which
+  Flux-applied manifests read the site's domain and ranges (e.g. postBuild substitution from a
+  site ConfigMap). Known trap: postBuild substitution blanks every undefined `${var}`, and some app
+  manifests contain unrelated `${...}` syntax (`infrastructure/cluster-settings/cluster-settings.yaml:6-9`).
+- For production the change is a no-op: rendered manifests and live objects stay identical. Prove it
+  (e.g. render before/after) per PR.
+- Changes reach production as reviewable, stacked PRs. Moving objects between pruned Kustomizations
+  needs the prune handover used in #1307/#1317. Never force-push.
+- The test may create its own entries in production's NetBird account and Cloudflare zone, limited to
+  its own names (owner, 2026-10-18). It must not modify or delete production's.
+- Egress containment stays: the test's Hetzner firewall blocks TCP to production's public edge
+  (`proxy.renner.uno`, `bootstrap.sh` `hcloud_apply_egress`). NetBird management, signal and relay run
+  on `netbird.renner.uno`, a different address, and stay reachable. Inbound traffic arrives through the
+  NetBird tunnel. The builder must check that the test's routing peer works with TCP to the proxy
+  hosts blocked (the proxy hosts' extra relays on TCP 33080 are then unreachable; UDP passes).
+- Stale production URLs in restored data stay as they are: Gitea's Woodpecker webhooks
+  (`https://ci.renner.uno/api/hook?access_token=…`, valid tokens), Home Assistant's `.storage`
+  (`external_url`, integrations), Immich's OIDC issuer in OpenBao `apps/immich`, WordPress post
+  content. They must not reach production: the test resolves its own names in-cluster
+  (k8s-gateway), and production names resolve publicly to the proxy, which the egress firewall
+  blocks. The builder verifies this holds for each of these.
+- Let's Encrypt: the test site uses Let's Encrypt staging (owner, 2026-10-18: "staging certs are
+  fine"), so test runs never count against the `renner.uno` limits production's renewals share (about
+  40 Certificates per run). The ACME server therefore becomes a site fact; today `ACME_SERVER` is in
+  the Flux-applied `infrastructure/cluster-settings/cluster-settings.yaml`, substituted into the
+  ClusterIssuer. The owner accepts browser warnings when inspecting the test.
+- To verify, not assumed: what the Cloudflare external-dns instance would write for the test names.
+  It runs `policy: upsert-only` with `--default-targets` set to a `*.cfargotunnel.com` name and
+  `--force-default-targets` (`apps/external-dns/helmrelease.yaml`); the public wildcard `*.renner.uno`
+  already resolves every test name to the proxy. A record it creates must not shadow the wildcard
+  with an unreachable target.
+- Passkey step-up: some apps require a passkey step-up after login (blueprint `passkey-stepup.yaml`:
+  Home Assistant, NetBird, Audiobookshelf, Immich, Gitea, Grafana). On the test site the owner would
+  register a new passkey after password login; the builder checks the flow allows that.
+
+## Decided (owner)
+
+- Subdomain of `renner.uno` as the test base domain.
+- Reachable from the internet through the NetBird proxy, like production; 1:1 with production
+  otherwise.
+- The test may create its own NetBird and Cloudflare entries, limited to its names.
+- Address ranges are site facts in the site files.
+- Only what must differ, differs.
+- Matrix out of scope; passkeys not a criterion, password login on the test.
+- The test's NetBird peers use their own rotator-minted setup key with access only where needed.
+- Features a site does not have are not applied there at all: their Kustomizations are suspended on
+  that site, so Flux does not try them and `flux get ks` shows them as suspended instead of failing
+  at a timeout ("noise", owner). The site file declares what the site has, like its other site facts:
+  the Intel iGPU, the IoT VLAN, and the UniFi gateway. Talos creates these as a ConfigMap (the way
+  `cilium-site` works), and Flux's root sync reads it to set `spec.suspend` on the affected
+  Kustomizations. Affected today: llama-server and wiki (iGPU), home-assistant-iot, matter-server and
+  music-assistant (IoT VLAN), external-dns-unifi (UniFi gateway; it does not run on the test
+  cluster). otbr is suspended on every site already. The builder confirms the list against the
+  current node-label gating (`hardware.renner.uno/intel-gpu`, `net.renner.uno/iot-vlan`, #1313); the
+  node labels stay, they say which nodes of a site carry the hardware.
+- Let's Encrypt staging on the test site.
+
+## Calls made in the design (not the owner's; check these)
+
+- Only the service CIDR becomes a site range; pod CIDR and LoadBalancer pool stay identical, because
+  with the restricted setup key nothing routes them between the sites.
+- The run-29 production ingress break was never explained. Run 30 turns on the parts that touch
+  production's public path, so the plan enables them in stages and checks production after each:
+  first `netbird-service-sync` for one pass (`ONCE`) against the test's own network, then the
+  routing peers, and stops on the first sign of production impact.
+- The builder checks whether the NetBird proxy accepts services under the test subdomain without a
+  separately registered custom domain (production services are custom domains `<app>.renner.uno`,
+  wiki `[[netbird-reverse-proxy-custom-domains]]`). If a one-time registration is needed, it is the
+  owner's step to approve.
+
+## Left to the builder
+
+- The exact test domain label and the test's service CIDR (must not overlap production's, the LAN, or
+  NetBird's ranges).
+- The substitution / site-value mechanism and how to keep it safe against unrelated `${...}`,
+  including on the root sync that sets `spec.suspend` (production must render unchanged: every
+  feature present, nothing suspended that is not suspended today).
+- How the test's NetBird setup key and router group are created, within the constraints.
+- How the Let's Encrypt limit is respected.
+- Order and split of PRs, and the restore-test run that proves it (run 30), including inspecting the
+  restored services through the internet.