homelab/kubernetes/RUNBOOK-cold-rebuild.md

Cold-rebuild runbook — homelab cluster from scratch

Rebuild the entire cluster from this repo + backups only, assuming nothing in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea gone).

Order: tofu → talos → flux → openbao → apps.

All three nodes are bare-metal MS-01 machines. Nothing provisions them: there are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi. Step 1 is therefore two unrelated halves — a tofu apply for the platform resources, and physically booting three machines from install media.

Someone has to be at the machines to boot them from USB — MS-01 has no IPMI and nothing here does PXE. Everything after that is remote.

A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes up on whatever untagged network its ports carry — not on management, which is a tagged VLAN on the X710 trunk. It is reachable there: that is the address apply-config --insecure talks to in Step 2, and it is how all three bare-metal conversions were driven. Find the leases on the UDM.

The circular-bootstrap problem this runbook exists to break

In steady state Flux's GitRepository flux-system pulls manifests from the in-cluster Gitea (https://gitea.renner.uno/konairius/homelab.git, kubernetes/clusters/homelab/flux-system/gotk-sync.yaml). But that Gitea is itself Flux-managed:

• its data lives on Ceph (Flux-deployed via Rook),
• its hostname resolves only through in-cluster external-dns + Traefik.

So Flux cannot reconcile without Gitea, and Gitea cannot come up without Flux having first deployed Ceph + ingress + DNS. In a cold rebuild that loop has no entry point.

The exit: bootstrap Flux from the external GitHub break-glass mirror github.com/konairius/Homelab instead of Gitea. That mirror already exists and stays current on its own: Gitea has a native push-mirror configured on the konairius/homelab repo (sync_on_commit=true, 8h interval fallback), so it holds last-good main with no cluster involvement — it is server-side inside Gitea, so it keeps tracking main independent of Woodpecker/agents. Nothing to set up; just confirm it before relying on it (Prerequisites). Once the cluster (and its in-cluster Gitea) is healthy again, switch the Flux source back to Gitea (Step 6).

Related docs: issue #187 (finding), kubernetes/apps/MIGRATION-gitea-woodpecker.md (the migration that created the circularity), kubernetes/MIGRATION-sops-openbao.md (the OpenBao/ESO secrets layer that Step 4 restores).

Prerequisites — off-cluster, verify BEFORE you start

These must exist outside the cluster or the rebuild cannot complete:

• This repo, checked out locally.
• GitHub mirror is current — github.com/konairius/Homelab at the main you intend to restore. Gitea's native push-mirror keeps it there on every push while Gitea is up; confirm the latest commit matches your intended recovery point (git ls-remote https://github.com/konairius/Homelab.git main, or the repo's Settings → Mirror Settings last-sync time in Gitea).
• GitHub PAT for flux bootstrap github, exported as GITHUB_TOKEN. flux bootstrap commits its sync manifests into the repo, so this needs write (classic: repo; fine-grained: Contents R/W on konairius/Homelab). This is the recovery operator's break-glass credential, unrelated to the server-side Gitea push-mirror.
• SOPS age key at ~/.config/sops/age/keys.txt. Still required: the flux phase creates the sops-age Secret, and any not-yet-migrated sops files decrypt with it. (Post-OpenBao-phase-4 this may be retired — see MIGRATION-sops-openbao.md.)
• OpenBao seal key, recovery key + root token — in the GPG pass store (homelab/openbao/seal-key, homelab/openbao/unseal-key — the former shamir key, now the recovery key — and homelab/openbao/root-token), and the GPG private key backed up off-machine. The seal key is the one the snapshots are encrypted under: without it, no OpenBao snapshot can be restored and no secret-consuming app can start.
• OpenBao raft snapshot reachable — nightly PVC snapshots (infrastructure/openbao/snapshot-cronjob.yaml) plus the Velero openbao-daily offsite copy. In a total-loss rebuild the offsite Velero copy is the one that survives.
• Tooling: talosctl, kubectl, flux, tofu, bao, plus the OpenTofu backend + UniFi credentials the opentofu/ stack needs. No Proxmox credentials — there is no Proxmox any more.
• Install media written and to hand: the Image Factory ISO named in Step 1, on a USB stick, for all three machines.
• Someone at the machines to boot them from USB. No IPMI, no PXE. A monitor is useful for reading boot errors but is not required to find the node afterwards — it DHCPs into management on its own (see the note at the top). Everything past the boot is remote.

Step 1a — tofu: DNS, object storage, UniFi

OpenTofu no longer provisions any node. The proxmox_vm module still exists in the tree but nothing instantiates it, so tofu apply here creates only the platform resources: UniFi DNS records including k8s.renner.uno and the node A records, the Hetzner object storage, and the NAT hairpin rules.

cd opentofu
tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=..."
tofu apply

Run this first regardless, because Step 2 needs k8s.renner.uno and the node FQDNs to resolve — the control-plane endpoint is baked into every generated config, and the nodes are addressed by name.

Step 1b — boot all three machines from install media

https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.10/metal-amd64.iso

Write it to a USB stick and boot each MS-01 from it. Each comes up in maintenance mode with a DHCP address on an untagged network — not on management, which is tagged. Read the leases off the UDM; those addresses are what Step 2 applies the first config to.

The schematic 0f6fb8bd… carries net.ifnames=1 — without it the predictable interface names that talos/patches/k8s-{1,2,3}.yaml match their VLAN parents against do not exist, and the node ends up with no address on any interface. Beyond that it carries:

• siderolabs/intel-ucode — no hypervisor supplies microcode here
• siderolabs/i915 — the Alder Lake-P Iris Xe. The Intel GPU device plugin (infrastructure/intel-gpu-plugin) turns this into the schedulable resource gpu.intel.com/i915, which apps/llama-server requests for LLM inference and which any hardware-transcoding workload would use too. Without the extension the plugin finds no /dev/dri and advertises nothing
• siderolabs/thunderbolt — inert until machine.kernel.modules requests thunderbolt/thunderbolt_net, present for the same reason. Note its udev rule auto-authorises any Thunderbolt device attached once those modules load
• no qemu-guest-agent, which has nothing to talk to on bare metal

Verify the disks before going further. Talos enumerates NVMe in an order that is not stable across boots, and the per-node configs select by serial for exactly that reason — the failure mode is the installer taking the Ceph OSD drive. talosctl -n <maintenance-ip> --insecure get disks should show three NVMe devices whose serials match the diskSelector and VolumeConfig entries in talos/patches/k8s-N.yaml.

Step 2 — talos: form the cluster + get kubeconfig

kubernetes/bootstrap.sh talos does this, including the CNI install that used to be a hand-typed step. It was unusable for months because it passed cilium-cni.yaml whole and the file's $patch: delete document failed against a fresh config (#581); that file no longer exists, so the failure is gone with it.

Verified as far as it can be without wiping a cluster: talosctl gen config with exactly the patches the script passes produces a config carrying cni: none and proxy: disabled that passes talosctl validate --mode metal. The apply-and-bootstrap half has not been exercised since the rewrite. Read the manual procedure below before running the script, so you know what it is doing and can take over at any step if it stops.

The procedure below is what actually works, and is the same one RUNBOOK-baremetal-conversion.md uses.

Generate the configs

This is a genuinely new cluster — nothing survives to copy secrets from, so generate them fresh. (In a partial rebuild, where one node still runs, pull secrets off it instead with talosctl get mc v1alpha1 + gen secrets --from-controlplane-config, or the new node forms its own cluster rather than joining.)

talosctl gen secrets -o secrets.yaml

for N in 1 2 3; do
  talosctl gen config homelab https://k8s.renner.uno:6443 \
    --with-secrets secrets.yaml --output-types controlplane \
    --config-patch @talos/patches/cluster.yaml \
    --config-patch @talos/patches/containerd-layer-retention.yaml \
    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
done

There is no CNI patch here any more. Talos supplies cni: none and proxy: disabled from cluster.yaml and nothing else; Cilium is installed in Step 2b below. See PROPOSAL-cilium-single-owner.md.

One thing the generator gets wrong for this repo, and it bites silently:

• Strip the generated HostnameConfig — it collides with machine.network.hostname from the per-node patch.

Validate each before applying:

talosctl validate --config install-k8s-$N.yaml --mode metal

Apply in two stages

Do not apply the full per-node config in maintenance mode. A document that references an interface which does not exist yet hangs the install: the node sits at STAGE: Installing indefinitely with KUBELET n/a and APISERVER n/a, and nothing times out. Concretely, the ens19 VLAN document must be removed from the first-stage config — its parent exists, but the install stalls anyway.

# Stage 1: minimal config, from the maintenance-mode address off the console
talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
# Stage 2: once the node is up on management, apply the complete config
talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml

--mode=try rolls back on its own if the node becomes unreachable, which is the only safety net there is for a network change that can strand a machine with no IPMI. Use it for every network-affecting apply, then make it permanent once proven.

Bootstrap etcd, then collect access

Bootstrap exactly one node:

talosctl -n k8s-1.srv.renner.uno bootstrap
talosctl -n k8s-1.srv.renner.uno kubeconfig .

scripts/import-cluster-access.sh is the supported way to get the kubeconfig into place afterwards.

The nodes will not reach Ready here. Talos is configured with cni: none, so there is no pod network until Step 2b — expect NotReady with container runtime network not ready and carry on.

Step 2b — cilium: install the CNI

Nothing schedules without it, Flux included, so this comes before Step 3. Talos no longer carries a rendered copy of Cilium; helmrelease.yaml is the only place it is configured, and this installs from those same values so the cluster comes up on what git describes rather than on a second, drifting copy. See PROPOSAL-cilium-single-owner.md for why it works this way.

bootstrap.sh talos already did this. What follows is the same thing by hand, for when the script stopped partway or you are not using it.

Every input is read out of helmrelease.yaml. Nothing here is a version or a URL you have to keep in step with anything — copy the block as it stands:

HR=kubernetes/infrastructure/cilium/helmrelease.yaml

helm repo add cilium "$(yq -r 'select(.kind == "HelmRepository") | .spec.url' "$HR")"
helm repo update cilium

yq 'select(.kind == "HelmRelease") | .spec.values' "$HR" > /tmp/cilium-values.yaml

helm install cilium cilium/cilium \
  --version "$(yq -r 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "$HR")" \
  --namespace kube-system \
  -f /tmp/cilium-values.yaml \
  --set prometheus.serviceMonitor.enabled=false

--version takes the HelmRelease's range verbatim; helm resolves it to the same release Flux would pick, so the two cannot diverge. That is the point of writing it this way — a pinned version in a runbook is a copy nobody maintains, and the next rebuild installs something the HelmRelease will immediately fight.

The one literal is serviceMonitor, disabled for this install only: its CRD arrives with the monitoring stack much later and the object would fail to apply. Flux adds the real one once it owns the release. Nothing else is overridden, which is what lets helm-controller adopt this release rather than replace it.

Ends when all three nodes are Ready.

Step 3 — flux: bootstrap GitOps from the GitHub mirror (break-glass)

This is where the circular bootstrap is broken. Because in-cluster Gitea does not exist yet, bootstrap Flux from the mirror by setting FLUX_SOURCE=github:

export GITHUB_TOKEN=<github-pat-with-write-on-konairius/Homelab>
FLUX_SOURCE=github kubernetes/bootstrap.sh flux

This runs flux bootstrap github --owner=konairius --repository=Homelab --branch=main --path=kubernetes/clusters/homelab --token-auth (see bootstrap.sh). Flux then:

1. installs its controllers + the sops-age Secret,
2. points GitRepository flux-system at the GitHub mirror,
3. reconciles infrastructure → Cilium, Rook/Ceph, external-dns, Traefik, cert-manager, OpenBao, External Secrets Operator, and eventually the in-cluster Gitea itself.

The phase waits for infrastructure, the Ceph cluster, and the Ceph object store to report Ready. Ceph mon/mgr/OSD bring-up on fresh disks is genuinely 5–10 min; the waits warn-and-continue rather than hard-fail, so read the output — a green step over a still-forming Ceph is a known trap.

Steady-state (non-rebuild) bootstrap is just kubernetes/bootstrap.sh flux with GITEA_TOKEN set; FLUX_SOURCE defaults to gitea.

Step 4 — openbao: restore the snapshot

bootstrap.sh flux does this step itself (restore_openbao): it creates the openbao-seal-key Secret from pass before Flux brings OpenBao up, initialises the empty server, restores the newest raft snapshot straight out of the Velero kopia repository, and waits until every replica has unsealed itself from the static seal. What follows is the same procedure by hand, for when the script is not an option.

OpenBao unseals itself from the static seal (openbao-seal-key Secret, key key, the 32-byte base64 key from pass show homelab/openbao/seal-key). The snapshots are encrypted under that key, so it must be the same key the backed up cluster used — a different one restores nothing. Until the snapshot is restored, the ClusterSecretStore stays NotReady, ESO cannot materialize Secrets, and every secret-consuming app stays Pending.

# 1. The seal key, before the pods start (a pod without it never starts).
kubectl create namespace openbao --dry-run=client -o yaml | kubectl apply -f -
kubectl -n openbao create secret generic openbao-seal-key \
  --from-literal=key="$(pass show homelab/openbao/seal-key)"

# 2. Wait for openbao-0 (uninitialised: 0/1 Ready is expected), then
#    initialise it. With an auto-seal, init hands out recovery keys and the
#    server unseals itself. These keys are throwaway: the restore replaces them.
kubectl -n openbao exec -it openbao-0 -- bao operator init \
  -recovery-shares=1 -recovery-threshold=1

# 3. Restore the most recent raft snapshot. Source it from the snapshot PVC
#    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
#    from the Velero `openbao-daily` offsite copy. Root token from the init
#    output. No -force: OpenBao checks the snapshot was written under this
#    server's seal key and refuses otherwise ("different autoseal key") — that
#    refusal is the signal the Secret is wrong. -force would skip the check and
#    leave a server that can never unseal.
kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
kubectl -n openbao exec -it openbao-0 -- \
  env BAO_TOKEN=<init root token> bao operator raft snapshot restore /tmp/restore.snap

# 4. The server seals against the snapshot's keyring and unseals itself from
#    the static seal; openbao-1 and openbao-2 follow (OrderedReady). Nothing
#    to type. Afterwards the tokens and recovery key from pass apply again.
kubectl -n openbao get pods -w

Verify the store recovers before moving on:

kubectl -n openbao exec openbao-0 -- bao status   # want: Seal Type static, Sealed false
kubectl get clustersecretstore openbao            # want: Valid / Ready
kubectl -n <ns> get externalsecret                # want: SecretSynced

A replica that stays sealed after a restart could not use the static seal — the Secret is missing or does not match the key the data was written under. Fix the Secret and restart the pod; bao operator unseal with the recovery key (pass show homelab/openbao/unseal-key) does not unseal an auto-sealed server and is only for recovery-mode operations. Details + rationale: MIGRATION-sops-openbao.md.

Step 5 — apps: let the rest reconcile

With Secrets flowing, the application layer converges:

kubernetes/bootstrap.sh wait-apps

Waits per-app (apps, authentik, monitoring, paperless, home-assistant, immich, etc.) and then waits for all cert-manager Certificates to issue. The volumes come back by themselves: each Velero Schedule has a restore-<schedule> Kustomization (clusters/homelab/restore.yaml) that applies a Velero Restore of the schedule's newest backup, restore-gitea first, and every app waits for its own. CNPG databases recover from barman (RECOVERY.md). Long tails are expected (immich's library). End-to-end health check: bash kubernetes/e2e-test.sh.

A Restore runs once and is never retried. One that ends PartiallyFailed, Failed or FailedValidation (other than "No backups found for schedule") keeps its restore-* Kustomization not Ready, and every Kustomization that depends on it stops applying, until the Restore is gone. The Restore of restore-<name> is velero/<name>. Read its result, fix the cause, then delete it; Flux applies it again against the newest backup:

velero restore describe <name> --details
kubectl -n velero delete restore <name>

Step 6 — switch the Flux source back to Gitea

The cluster now runs on the GitHub mirror, but the in-cluster Gitea is healthy again (Flux brought it up in Step 3, restored per MIGRATION-gitea-woodpecker.md if its PVC was empty). Return the source of truth to Gitea so normal push→Flux→mirror flow resumes:

export GITEA_TOKEN=<gitea-pat>
kubernetes/bootstrap.sh flux        # FLUX_SOURCE defaults to gitea; re-runs
                                    # `flux bootstrap gitea`, re-pointing
                                    # GitRepository flux-system at Gitea.

Confirm kubectl -n flux-system get gitrepository flux-system again shows the gitea.renner.uno URL and reconciles. Gitea's push-mirror then resumes tracking main on the next push automatically — nothing to re-enable.

Mirror maintenance — none required

Keeping the break-glass source current is already handled by Gitea itself, not by any cluster workload. A native push-mirror is configured on the konairius/homelab repo (remote https://github.com/konairius/Homelab.git, sync_on_commit=true, 8h interval fallback), so Gitea pushes main to GitHub on every git receive for as long as Gitea is up — which is also the only time main advances. It runs server-side inside Gitea, so it survives Woodpecker / agent outages, and it predates and survived the in-cluster migration.

There is no setup step and no credential to provision here. To inspect or audit it: repo → Settings → Repository → Mirror Settings in Gitea, or GET /api/v1/repos/konairius/homelab/push_mirrors (admin token) — check last_error is empty and last_update is recent.

Recovery-point & known gaps

• The mirror is only as fresh as the last healthy push before the incident; in-cluster commits made after Flux/Gitea broke are not on the mirror (Gitea couldn't push them if Gitea itself was down). This is the intended recovery point, but be aware of it when reconciling drift.
• Booting is physical; the rest is not. MS-01 has no IPMI and nothing here does PXE, so someone has to put a USB stick in three machines. From there it is remote — a maintenance-mode node DHCPs untagged and is reachable at that address, which is how all three bare-metal conversions were driven. #576 claims a freshly booted node "comes up unreachable and must be installed from a console". That is not what happens and has never been observed here.
• bootstrap.sh talos is dead — it cannot generate a config (#581), and Step 2 no longer uses it. Its flux and wait-apps phases are still used in Steps 3, 5 and 6 and work; do not read this as the whole script being gone. Its flux phase carries the OpenBao restore (Step 4); the seal key and the other secrets are read from pass or streamed in at run time and never live in CI.
• OpenBao unseals itself, but only under the seal key the snapshot was written with. Three raft replicas mean the service survives losing one; a sealed server is neither a voter nor a standby. A pod that stays sealed after a restart has the wrong openbao-seal-key Secret — Step 4 covers this; it is repeated here because it is the step most likely to be half-done under pressure.
• Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts IPAM, not host networking. Nothing in the rebuild needs special handling for it — the nodes hold no address on that segment — but a rebuilt cluster will not have the whereabouts allocations from before, which is harmless since they are assigned fresh.
• warn-and-continue waits in the flux phase can mask a genuinely broken Ceph/infra bring-up; always read the phase output rather than trusting the exit code.