homelab/kubernetes/RECOVERY.md

CNPG disaster-recovery runbook

Closes the gap in issue #190. WAL + base backups are archived continuously to Hetzner Object Storage via the barman-cloud plugin, and every backed-up cluster below (like wp-pg) keeps bootstrap.recovery in git: a cold rebuild restores each from its backup with no manual step. This document covers the precondition, watching and verifying that restore, and the reasoning behind it.

Cluster	Namespace	app DB / owner
ak-pg	authentik	authentik/authentik
im-pg	immich	immich/immich
pprl-pg	paperless	paperless/paperless
ha-pg	home-assistant	homeassistant/homeassistant
mx-pg	matrix	synapse/synapse
tr-pg	trmnl	terminus/terminus
wp-pg	woodpecker	woodpecker/woodpecker

Each cluster <x> has two ObjectStores in kubernetes/cnpg-object-stores/: <x>-store, where it archives WAL and takes base backups, and <x>-store-source, which bootstrap.recovery reads. Their paths come from the flux-system/cnpg-paths ConfigMap, which bootstrap.sh writes once per cluster incarnation: the source is the prefix in s3://renner-uno-k8s-cnpg whose <x>/base/ holds the newest base backup, the archive a new prefix <x>-g<timestamp>. The barman serverName is the cluster name in both. gf-pg (monitoring) has no ObjectStore and stays on initdb.

The current prefixes: kubectl -n flux-system get cm cnpg-paths -o yaml (CNPG_<X>_SOURCE, CNPG_<X>_ARCHIVE). bootstrap.sh never rewrites an existing cnpg-paths. Adding a CNPG cluster to a running cluster therefore means adding its two keys to that ConfigMap by hand, with the archive prefix as the source as well; without them cnpg-object-stores fails to build and every app Kustomization behind it stalls.

---

Why the recovery stanza is safe on a live cluster

CNPG's spec.bootstrap is honoured only when a cluster is first initialised (empty PGDATA, no existing PVCs). On every subsequent reconcile of a live cluster the field is inert — CNPG never re-bootstraps, adopts existing PVCs as-is, and changing bootstrap on a running cluster is a no-op. The recovery stanza therefore cannot clobber a live database. It only acts when PGDATA is genuinely absent — exactly the cold-rebuild case it covers.

The sharp edge: bootstrap.recovery has no initdb fallback. If the object store is empty (a brand-new cluster that has never been backed up, or a destroyed bucket), a recovery bootstrap hangs forever instead of creating an empty DB. That "fail loud" behaviour is the chosen trade over initdb's "silently come up empty".

Historical note: bootstrap.recovery from barman was the original seed design (iteration 2). It was temporarily replaced by initdb + a one-shot pg_dump | pg_restore seed Job for the self-hosted-to-cluster migration. Those seed Jobs were deleted post-cutover (2026-05-09), which is what left initdb behind as a no-op-on-rebuild leftover, until the clusters above were switched back to recovery (2026-10-15).

---

Cold-rebuild recovery (per cluster)

Example uses ak-pg; substitute the row from the table above for the others.

1. Precondition — confirm a backup actually exists

Never start a recovery you can't finish. Check the object store has a base backup and recent WAL for the cluster's serverName before touching anything. Backups live under <prefix>/<serverName>/base/, one prefix per incarnation (ak-pg and ak-pg-g<timestamp>):

# From a machine with the Hetzner S3 creds (see hetzner-creds-externalsecret):
aws --endpoint-url https://fsn1.your-objectstorage.com \
    s3 ls s3://renner-uno-k8s-cnpg/ | grep ak-pg
aws --endpoint-url https://fsn1.your-objectstorage.com \
    s3 ls s3://renner-uno-k8s-cnpg/<prefix>/ak-pg/base/

If that is empty, stop — recovery will hang forever (no initdb fallback). The data is gone; this is a restore-from-nothing situation, not a rebuild.

2. Watch it recover

Flux applies cluster.yaml as part of the app's Kustomization; its bootstrap.recovery + externalClusters stanza (serverName = cluster name) drives the restore.

• Recovery is a physical restore (latest base backup + WAL replay), so it brings back all databases and roles, not just the app DB. The database/owner fields only tell CNPG which app user the -app service should target after promotion.
• After the restored primary is promoted, spec.plugins[].isWALArchiver: true archives to <x>-store, the new incarnation's own empty prefix; the prefix it recovered from is left untouched. Its first base backup there is taken right away (immediate: true on the ScheduledBackup).

kubectl -n authentik get cluster ak-pg -w
# phase progresses: "Setting up primary" -> "Creating a new replica"
#                   -> "Cluster in healthy state"
kubectl -n authentik logs -l cnpg.io/cluster=ak-pg,role=primary -c postgres --tail=50

3. Verify the data

kubectl -n authentik exec -it ak-pg-1 -c postgres -- \
  psql -U postgres -d authentik -c '\dt' -c "select count(*) from authentik_core_user;"

Confirm row counts look sane and the app boots against the restored DB.

---

Velero backup verification (backup-verify)

Monthly (1st, 08:00 UTC) CronJob velero/backup-verify. It creates a throwaway Hetzner Cloud VM in fsn1 (project homelab-backup-verify) and runs kopia snapshot verify --verify-files-percent=100 read-only against the latest snapshot of every DataUpload of each schedule's newest finished backup. Snapshot sizes are checked against DataUpload.status.progress.totalBytes. Nothing is restored into the cluster. A full run takes about an hour (2026-10-10: 39 PVCs, ~410 GB, 61 min, cx23).

• Results: kubectl -n velero logs job/<backup-verify-...>; FAIL lines name the backup, PVC, snapshot and cause.
• A failed run fires KubeJobFailed (velero).
• Manual run: kubectl -n velero create job bv-manual --from=cronjob/backup-verify; limit it with env ONLY_NAMESPACES=apps,gitea.
• Every run first deletes everything labelled managed-by=backup-verify in the Hetzner project, so a leftover VM is cleaned up by the next run. Do not start a manual run while another one is running: its sweep deletes the other run's VM.
• This checks data readability, not a full restore. Disaster-recovery rehearsal is separate.

---

CNPG restore-test (verify backups are restorable — non-destructive)

The monthly Velero backup verification (kubernetes/infrastructure/velero/) checks the Velero PVC backups but does not cover CNPG (barman-cloud is a separate code path). Run this drill periodically to prove the CNPG backups are actually restorable, without touching the live clusters. It restores into a throwaway cluster in the same namespace (so it can reference the existing namespaced ObjectStore and creds) and declares the barman plugin with isWALArchiver: false — loaded so it can read WAL to replay during the restore, but never archiving, so it only ever reads the backup path.

NS=authentik ; SRC=ak-pg ; STORE=ak-pg-store ; DB=authentik
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ${SRC}-rtest
  namespace: ${NS}
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:16   # match the live cluster's image
  storage:
    # Delete-reclaim class, safe here BECAUSE this is the non-destructive
    # restore-*test* — a throwaway verification cluster, never a real DR target
    # (a real cold-rebuild recovers the live cluster on its own Retain
    # ceph-block storage). So its volume SHOULD vanish on teardown rather than
    # leak a Released PV + orphaned RBD image.
    storageClass: ceph-block-ephemeral
    size: 10Gi
  # The barman plugin MUST be declared so the instance manager can load it to
  # replay WAL during the restore. Omitting it (as an earlier version of this
  # runbook said to do) fails at instance start with "Unknown plugin:
  # barman-cloud.cloudnative-pg.io" and the cluster never becomes healthy.
  # isWALArchiver: false keeps it read-only — loaded for restore, but it never
  # archives, so it only ever READS the backup path (verified 2026-07-25 on im-pg).
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      enabled: true
      isWALArchiver: false
      parameters:
        barmanObjectName: ${STORE}
  bootstrap:
    recovery:
      source: ${STORE}
      database: ${DB}
      owner: ${DB}
  externalClusters:
    - name: ${STORE}
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: ${STORE}
          serverName: ${SRC}
EOF

kubectl -n ${NS} wait --for=jsonpath='{.status.phase}'='Cluster in healthy state' \
  cluster/${SRC}-rtest --timeout=20m
kubectl -n ${NS} exec ${SRC}-rtest-1 -c postgres -- psql -U postgres -d ${DB} \
  -c "select count(*) as user_tables from pg_stat_user_tables;" \
  -c "select pg_size_pretty(pg_database_size('${DB}')) as db_size;"
# NOTE: pg_stat_user_tables row estimates (n_live_tup) read 0 immediately after
# a restore — the stats collector hasn't run yet, so that view is NOT a data
# check. Confirm real data with pg_database_size above, or count(*) on a known
# table (e.g. im-pg: `select count(*) from asset;` returned 33316 on 2026-07-25).

# Tear down. Because the cluster above uses the ceph-block-ephemeral
# (reclaimPolicy: Delete) class, deleting it removes its PVC -> PV -> RBD image
# automatically — no manual PV reclaim needed (the old Retain dance leaked
# orphaned images; issue #262).
kubectl -n ${NS} delete cluster ${SRC}-rtest

For im-pg use imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration and add its shared_preload_libraries (vectors.so, vchord.so) so the restored vector extensions load.

Automating this as a weekly CronJob (mirroring the Velero one) is the logical next step once a first supervised run confirms the mechanics — tracked in #190.