homelab/kubernetes/MIGRATION-sops-openbao.md

Migrating k8s secrets: SOPS/age in git → OpenBao + External Secrets Operator

Goal: secret values leave git. Git keeps only ExternalSecret manifests (non-sensitive references); the External Secrets Operator (ESO) syncs real values from OpenBao into k8s Secrets at reconcile time. Editing a secret becomes bao kv put / UI edit instead of the sops encrypt-commit-push cycle.

Scope: the 31 *.sops.yaml files under kubernetes/ only. The Ansible side (ansible-vault via GPG pass) is a separate mechanism and stays untouched; unifying it onto OpenBao is a possible follow-up, not part of this migration.

The two constraints that dictate everything

1. Never commit a CR that cannot become Ready into a wait: true Kustomization. A ClusterSecretStore pointing at an uninitialized OpenBao stays NotReady and freezes the whole Flux dependsOn chain — the exact shape of the 2026-07-13 ImageUpdateAutomation incident. Hence the strict phase split below: the store CR is only committed after OpenBao is initialized and its k8s auth verified by hand.
2. The age key is retired LAST. While it lives, any migrated app rolls back with a plain git revert (sops file returns, decryption still configured). Retiring the key (phase 4) is the point of no return — everything before it is cheaply reversible.

DR chicken-and-egg to accept consciously: after migration, a cluster rebuild needs OpenBao's data (raft snapshot restore + unseal) before any secret-consuming app can start. See "Disaster recovery" below — the runbook must exist before phase 3 begins, not after.

Phase 1 — deploy (this PR)

Adds kubernetes/infrastructure/openbao/ (single-node raft, 5Gi ceph-block, UI at openbao.renner.uno, not publicly routed) and kubernetes/infrastructure/external-secrets/ (ESO, CRDs via CreateReplace). No ClusterSecretStore yet (constraint 1). Merging is safe: nothing consumes either component; the OpenBao pod sits 0/1 Ready (sealed) by design (disableWait on the HelmRelease keeps helm-controller green).

Post-merge, one-time init (single admin → 1 share / threshold 1):

kubectl -n openbao exec -it openbao-0 -- bao operator init \
  -key-shares=1 -key-threshold=1
# Store BOTH outputs in the GPG pass store, nowhere else:
# (bootstrapped 2026-07-13: key 6B8FED8020C1478F892EA83816866E146A8252A4,
#  no passphrase yet — set one via `gpg --edit-key ... passwd`, and BACK THE
#  KEY UP off-machine: without it, pass and thus unseal key are gone.)
pass insert homelab/openbao/unseal-key
pass insert homelab/openbao/root-token
kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key

Unseal is automatic since 2026-09-08 (static seal, PR #825, issue #805): the seal key lives in the openbao-seal-key Secret (created by bootstrap.sh from pass show homelab/openbao/seal-key) and every restarted replica unseals itself. The shamir key above became the recovery key in the migration; it is kept in pass under its old name, homelab/openbao/unseal-key. The server runs three raft peers (see infrastructure/openbao/helmrelease.yaml); a sealed one is neither a voter nor a standby, so a replica that stays sealed after a restart — wrong or missing seal-key Secret — costs redundancy, two cost quorum. The trade-off taken with the static seal: the key that unlocks the store sits in the same etcd as the Secrets ESO already syncs there in plaintext. Symptom of sealed: ESO stores go NotReady, synced Secrets keep their last value (they are real k8s Secrets — pods keep running).

Phase 2 — configure + ClusterSecretStore (follow-up PR)

Manual, with the root token (then stop using the root token day-to-day):

export BAO_ADDR=https://openbao.renner.uno
bao login   # root token, from pass
bao secrets enable -path=homelab -version=2 kv
# NOTE: `bao audit enable` is disabled in OpenBao 2.5 — audit devices are
# declarative-only, configured in the server HCL (see the audit stanza in
# infrastructure/openbao/helmrelease.yaml; keep log_raw OFF or plaintext
# secrets land in Loki).
bao auth enable kubernetes
bao write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc.cluster.local:443
bao policy write eso-read - <<'EOF'
path "homelab/data/*"     { capabilities = ["read"] }
path "homelab/metadata/*" { capabilities = ["read", "list"] }
EOF
bao write auth/kubernetes/role/eso \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=eso-read ttl=15m

(auth/kubernetes/config from outside the pod needs the reviewer JWT flow; simpler: run these bao commands via kubectl exec in openbao-0, where the in-cluster host/CA defaults apply.)

Optional in the same session: OIDC auth method against authentik (issuer https://auth.renner.uno/...) for human UI login, so the root token goes back into pass and stays there.

Then the follow-up PR adds to kubernetes/infrastructure/post-crds/:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: openbao
spec:
  provider:
    vault:                       # ESO's provider name; speaks OpenBao fine
      server: http://openbao.openbao.svc.cluster.local:8200
      path: homelab
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

Merge only after bao read auth/kubernetes/role/eso looks right; verify the store goes Ready before starting phase 3.

Phase 3 — migrate, app by app

Per sops file: put the values into OpenBao, replace the manifest, one PR per app (or small batch), verify the app's Secret + workload before the next.

# 1. decrypt and read (from repo root; .sops.yaml lives under kubernetes/)
sops -d kubernetes/apps/evcc/evcc.sops.yaml
# 2. write values (key per Secret data field)
bao kv put homelab/apps/evcc username=... password=...
# 3. replace evcc.sops.yaml with evcc-externalsecret.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: evcc                    # target Secret keeps its old name — pods
  namespace: evcc               # and mounts don't change at all
spec:
  refreshInterval: 1h
  secretStoreRef: {kind: ClusterSecretStore, name: openbao}
  target:
    name: evcc                  # must match the old Secret's metadata.name
  data:
    - secretKey: username       # key in the k8s Secret
      remoteRef: {key: apps/evcc, property: username}
    - secretKey: password
      remoteRef: {key: apps/evcc, property: password}

# 4. update the dir's kustomization.yaml resource entry, PR, merge,
#    then verify the real consumer path:
kubectl -n evcc get externalsecret evcc   # SecretSynced/Ready
kubectl -n evcc get secret evcc -o yaml   # keys present, app still works

When the last sops file in a Flux KS path is migrated, drop the spec.decryption block from that KS in kubernetes/clusters/homelab/ in the same PR — a decryption block with nothing to decrypt is dormant drift.

Suggested order — blast radius ascending, checklist:

Warm-up (infrastructure, low blast radius): — done 2026-07-13

• infrastructure/external-dns-unifi/secret.sops.yaml
• infrastructure/netbird-ingress/secret.sops.yaml
• infrastructure/velero/secret.sops.yaml + hetzner-creds.sops.yaml
• infrastructure/post-crds/cloudflare-secret.sops.yaml (decryption blocks already dropped from infrastructure + infrastructure-crds — these two KS paths are sops-free)

Apps, non-critical: — done 2026-07-13

• apps/evcc/evcc.sops.yaml
• apps/otbr/dataset-secret.sops.yaml
• apps/libation/bootstrap-secret.sops.yaml (binary SQLite payload — stored base64 in KV, ES uses decodingStrategy: Base64)
• apps/external-dns/cloudflare-secret-apps.sops.yaml
• apps/monitoring/ (4: alertmanager-smtp, grafana-oauth, loki-basic-auth, pve-exporter-config)
• apps/woodpecker/secret.sops.yaml (CI must keep building — verify a pipeline run after) (decryption blocks dropped from apps, evcc, otbr, libation, monitoring + the dormant one on grafana-db)

Apps, stateful/central: — done 2026-07-13

• apps/home-assistant/ (5: backup-password, config-sync-token, mosquitto-evcc-credentials, prometheus-token, shelly-ble-proxy-key)
• apps/immich/secrets.sops.yaml
• apps/paperless/secrets.sops.yaml + ftp-scanner-secret.sops.yaml (decryption blocks dropped from home-assistant, immich, paperless
  • the dormant one on speech-to-phrase)

Seed (bootstrap-only; seed KS is suspended — migrate the files anyway so a future rebuild doesn't need the age key): — done 2026-07-14

• seed/authentik/, seed/immich/, seed/paperless/ barman-creds + hetzner-creds (6), seed/influxdb/bootstrap-secret.sops.yaml (decryption blocks dropped from all seed-* KS incl. the dormant ones on seed-audiobookshelf/seed-music-assistant and the suspended seed KS)

Last, the identity provider (everything else must be proven first): — done 2026-07-14

• apps/authentik/secret.sops.yaml + samba-bridge-secret.sops.yaml (authentik KS decryption dropped — ZERO sops files remain under kubernetes/; phase 4 may start once the soak criteria are met)

Phase 4 — decommission SOPS (point of no return)

Backup is decoupled from the soak gate below and should happen now — it only adds an offline copy in pass, it doesn't touch the live decryption path, so there's no reason to wait for the reconcile/unseal soak:

• Back up the live age key to pass: pass insert -m homelab/sops-age-key < ~/.config/sops/age/keys.txt (verify: pass show homelab/sops-age-key | diff - ~/.config/sops/age/keys.txt exits 0). pass's GPG key already has an external backup (see Disaster recovery below), so this gives the age key the same offline durability as the OpenBao unseal key/root token.

Only when every box above is checked and the cluster has survived at least one full reconcile + one OpenBao pod restart/unseal cycle:

• Remove remaining spec.decryption blocks from all Flux KS.
• Delete kubernetes/.sops.yaml; delete the sops-age Secret in flux-system.
• Remove SOPS_AGE_KEY from Woodpecker CI secrets.
• Re-label the pass backup from homelab/sops-age-key to homelab/sops-age-key.retired (pass mv homelab/sops-age-key homelab/sops-age-key.retired) — git history stays decryptable.

Disaster recovery (write into ops docs before phase 3)

• Backups: ✅ nightly raft-snapshot CronJob (03:00, keeps 7 on a dedicated PVC; see infrastructure/openbao/snapshot-cronjob.yaml) + velero openbao-daily Schedule (06:15) ships that PVC offsite, 30 d retention. A raft snapshot restore needs the matching seal key — which lives in pass, outside the cluster (GPG key backed up externally 2026-07-14). A failed snapshot Job is the sealed-after-restart alarm. Still owed: one tested restore before phase 4.
• Cluster rebuild order: seal-key Secret from pass → Flux brings up openbao (empty) → init → restore raft snapshot → replicas unseal themselves → ClusterSecretStore Ready → ESO materializes Secrets → apps start (bootstrap.sh flux does all of it; RUNBOOK-cold-rebuild.md Step 4 by hand). Until then, secret-consuming apps simply stay Pending on missing Secrets — same as any missing dependency.
• Break-glass: pass holds seal key, recovery key + root token (GPG, offline-capable). With those, everything is recoverable from a raft snapshot alone.
• Ansible Vault password: ansible/vault-password.sh reads ANSIBLE_VAULT_PASSWORD from the environment with no offline backup as of 2026-07-14. It gates decryption of several Ansible-Vault files (PBS credentials on nas-1.renner.uno, plus edge_host, grist, authentik, paperless, rustfs vars) — unrelated to the SOPS/OpenBao migration but in the same recoverability class. Back up with pass insert homelab/ansible-vault-password; verify with ANSIBLE_VAULT_PASSWORD=$(pass show homelab/ansible-vault-password) ansible-vault view ansible/inventory/host_vars/nas-1.renner.uno/vault.yml decrypting cleanly.

Known gotchas

• ESO chart pinned >=2.7.0 <3.0.0; major bumps have moved CRD APIs before — bump together with all ExternalSecret/ClusterSecretStore manifests.
• refreshInterval is the propagation delay after a bao kv put; force with kubectl annotate externalsecret <n> force-sync=$(date +%s).
• The OpenBao HelmRelease has disableWait — a genuinely broken chart upgrade will NOT be caught by helm-controller; watch the pod after chart bumps.
• Sealed OpenBao ≠ outage: existing k8s Secrets persist; only changes stop propagating until unsealed.