fix(monitoring): privileged is the mechanism, not the shortcut

Third attempt, and the first two were wrong for the same reason: I kept looking for the gate in the capability and syscall layers, and it was never there. Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything else dropped. It found all three drives per node and could open none of them. Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the default action for a syscall an unlisted profile blocks and that the kubelet runs seccompDefault: true. /proc/1/status confirmed both capabilities effective and a filter loaded, so the theory was at least testable — and removing the filter changed nothing. What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1` inside the container also fails with EPERM. Reading a block device involves no ioctl, no capability and nothing a seccomp profile filters. The denial is the device cgroup, which containerd applies to every non-privileged container regardless of whether /dev is mounted. Mounting /dev makes the nodes visible; it does not make them usable. Kubernetes has no per-device allowlist in a Pod spec — no equivalent of `docker --device`. So the honest position is that `privileged: true` here is not a lazy shortcut standing in for a narrower setting; it is the only mechanism the API offers, short of writing a device plugin. The manifest now says that, and records what was ruled out and how, so the next reader does not spend the same two rounds in the capability layer. The /dev mount loses its readOnly flag with it. smartmontools opens NVMe devices read-write on some paths, and once the container is privileged that flag suggests a boundary that is not there. A device plugin advertising the drives as a schedulable resource is the genuinely narrower answer and is noted as such. It is worth building when more than one workload needs raw device access; for a single read-only metrics exporter it is a component to write, deploy and maintain in exchange for the same access. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'fix(monitoring): unconfine seccomp so smartctl can open the devices' (#670 ) from smartctl-seccomp into master
2026-08-27 15:14:32 +02:00 · 2026-08-27 13:05:08 +00:00 · 2026-08-27 15:03:34 +02:00 · 2026-08-27 12:56:00 +00:00 · 2026-08-27 14:42:18 +02:00 · 2026-08-27 12:25:39 +00:00
55 changed files with 2491 additions and 610 deletions
@@ -101,7 +101,7 @@ steps:
        bash scripts/ci/make-diag-kubeconfig.sh

  - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:

 steps:
  - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/ansible
    depends_on: []
    # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-ansible:latest

  - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/k8s
    depends_on: []
    when:
@@ -77,7 +88,7 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-k8s:latest

  - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/tofu
    depends_on: []
    when:
@@ -108,7 +119,7 @@ steps:
  # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
  # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
  - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: secret-rotator
    depends_on: []
    when:
@@ -137,7 +148,7 @@ steps:
  # container, but nothing about it is zigbee-specific. Exists so tools come from
  # the image rather than an `apk add` on every container start.
  - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-jq
    depends_on: []
    when:
@@ -166,7 +177,7 @@ steps:
  # critical path of every node rebuild. The download now happens here, once,
  # where the sha256 is checked — see the Containerfile for the full rationale.
  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/cni-plugins
    depends_on: []
    when:
@@ -195,7 +206,7 @@ steps:
  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
  # base, so this one needs an image rather than a busybox substitute.
  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-openssl
    depends_on: []
    when:
@@ -224,7 +235,7 @@ steps:
  # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
  # builds the same way and lands in the same registry so the audit sees it.
  - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/heating-can
    depends_on: []
    when:
@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
      - |
        set -eu
        mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
        NEW=$$(npm view @anthropic-ai/claude-code version)
        case "$$NEW" in
          [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
  #    --build-arg to the resolved version (not "latest") guarantees the tag and
  #    the image content agree even if a new release lands mid-run.
  - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/claude
    depends_on: [decide]
    environment:
@@ -176,13 +186,10 @@ steps:
          echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
          exit 0
        fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
        # Exact-string swap of the current tag for the new one, consumers only.
        for f in $$CONSUMERS; do
          sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
        {
          printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
          printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
          printf 'run the image so CI stops running a stale Claude Code.\n\n'
          printf 'Merging is a human decision — review, then merge to roll it out.\n'
        } > /tmp/pr-body.md
@@ -17,6 +17,6 @@ when:

 steps:
  - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    commands:
      - sh scripts/ci/test-model-selection.sh
@@ -25,7 +25,7 @@ steps:
    # fetch-models.sh response mapping. No cluster, no secrets, no network — the
    # ci-claude image just provides sh + jq. Keeps the flag-injection regression
    # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    commands:
      - sh scripts/ci/test-model-selection.sh

@@ -62,7 +62,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -65,7 +65,7 @@ when:

 steps:
  - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -42,7 +42,7 @@ steps:
    # this step is skipped — otherwise we'd fork-bomb PRs.
    when:
      - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -30,7 +30,7 @@ when:

 steps:
  - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -0,0 +1,13 @@
+$ANSIBLE_VAULT;1.1;AES256
+32373134633930343261336362373335623735656634313034393731393661643361346634623439
+3561363963333533313264643036663731656465343162330a333964383462373934303766323533
+63613065313063626261383236396363656565623538666565646661353766643133306334396561
+3961376234623236620a616339326238366538333666316538396465666365373834313064653963
+38623233616633383435373836613438333763343932636239386263306238376137643531373161
+35386437313465643437343832623935353238316565366537646530323233363837656631383962
+32653332333162623966366163633732356637323461613966313263313066643966373335313162
+30306362346561633566316166313164323635356363643034663933363464663562343136646137
+66643536363734316533663032643637356136363237376365613330633961616665323430633135
+36323733633464306330666232663433616533373466633939353061663136666665323432373666
+36393930333435333238306238373931393232393861333334643139306138653336633534626362
+61323038303838396438
@@ -1,201 +1,29 @@
 $ANSIBLE_VAULT;1.1;AES256
-32666334326535346466653862343263373365346261313136653366366664633436356536383437
-6666326436366433373163353262383031333264633530390a346539316165633939346663623264
-30653839376635626261323234326539313235623962306639396633343533363561376363363864
-3165323130633061350a346233396530303737316361653337643264383230613635383932636133
-61336561663264393831313364366665636166306437383835303332643934393062623163343861
-31643063373432643531663538666363613331626564376539336435343161333237333036343664
-31636635316631376563326637303337386337313330323838666562303463363361346531613363
-64383062396132373536663832356266383430316465643934636537396466366335636131623339
-34316166323730316463666465613734383339636264306431623535656362633739386362393134
-30343837363361333666616332646131663738663737663434303437336130313133306236313739
-36363731386163353136376230393262653334366239653032613034323333386433666262353364
-30326364343735643739626563393661616238313265633566356639623639383138303934653737
-64356139656163666263346539663733356239386333313763653434343163353436323937663034
-35653236616135383465656536323738346662396366653435383337383233643232656435356335
-63616138666662386337323865356439303961396138613965303465373766343939333933316664
-35356561353366663361306336303339633362623332386330383533336461306139613232306363
-62613731613835336338346236373636343463383433613938393865666165653935353830633966
-61393266623961343665343136323235393764313461306138343334366631313531396663666136
-34636436666238376138633461636438356461643461356139313437393736666338623038353933
-34633032653930623065623639646231346133343231383032663438326164313136613361356232
-38306565353737623532653231303963616336323563663137316139303136623234663432373661
-37333431663233656635373433323461663164303137633831306232626465303761633934336461
-65613534633665663565366634623362303964313862333936383032363137306336636563383435
-35366666313065393463656430303436346334313930616166383861613830373962353536303866
-39636165393861363161633734343838346338303036306639316231313738383831626639303833
-62316336623135336462633439323533383931666563643839383236636134363466383561633363
-64366466336362376234336463383639323165306631633437333639333263373837666565306163
-34666162336166623831353436363831666561376562613732333938353638333134323133346435
-61663261373433333635643761373465653031303236613465643931356433656161373463316331
-34326163633037633737393338313931353130306134663839623332666365373663643330353864
-36366535313032333262373038333161336230303165646133303261333839376432393435613434
-39633830316236653962663464363637333433393665353066653230383339666130653135343332
-32623165386535323832626536316665396564623064633564663437613930313665616163326637
-61363538333761633264396430326539643833313830346565643937656137376464646664313636
-38626165306661353631393031383063646463363031633435373735653838636634316634323738
-38313832376265616631623761393666346133636135666664643661646130306237383134383934
-39643561363962323261343236393630656435653261633933656262623737613130353531386438
-39643335306436636238393734646238653465626232643161316539346435353130633138393863
-63343965346361353036646561636339303063393033633162666639303166303265333233386137
-38643835616131303263366539646130643630366139623032366532653031303336323463326263
-37313566633032313931313565616131376635643030373833323838393633653531653664333032
-33623566333731636462366133653837636238383832353362653230393164343135663062373438
-61613832336633343038373565316431643432626532636435653438383536336166383436393035
-66613037663739616337383136663034643433333933643862396330356533636439666536353338
-30313761633961383337353030333862396164376337393338303565663963633662633134653362
-30346563643537323431313137636364623231376364386364633739313438376663343339333330
-64623663303535393262376437626637316532643139353262393231376135663161663862646666
-65303463393966393633613964353132303863373565353131396435623634336364313864373465
-61376335313830383534643431383966616565346466633633613933643231303732373938313434
-38383731653938656331646239353330666266336566633232353835653239393232366634663165
-31313766643539373133396563633637336133633034633236393336663064626630376632336264
-63346136613165613063316264646561633430613630396263333562366430326432323264383233
-30616631616535666631303838326330363339343463303831376531396164333539386263343334
-64313363626363316266343735313263613038383464396564313336373831393936336237303361
-65346161323961336631366263353161616562633533323036383366613637333832323466326535
-36376661346133383461626461646237373531333532656239333566323433333139633966343465
-36653130646636336534353331623062313863666637306462343139333031313234633636613232
-64383663626566616465623938346531623362383839316361366566336166386461336634303339
-63363931303866376462303634633033386332373564393932616638323764383630643839393630
-38373533636561613131343138383765376332336434303930326630303735313462316666333861
-37346336613339643466636665326466386664376438646666363537306533303036613135303230
-65303733366163363964613135306538393234636262326639613439363063326237306337666564
-32303161373661633231333662303565383134333737343530616631363337396333346261646636
-38663439653435623236366437643334393432383636316664306530303133396561396435313936
-38316637393365343936636438666461326436353636323537343363623337656536613662393333
-32346638636532333266323036623932316265303134633265366634623336386138653165643662
-35396637636136386366663762346236343565623133623336363530653039363166383534313632
-39333162396534376335646664383362366663373531646132643462396131333932653464326532
-39326633616335633030373463303665383261323763303863383566386536646136613362323034
-35313064313533616262666439663266353136616466643562666131366636623166333030346333
-62353539396131666434356238646439326337343239366234383866613065636232613863323931
-38363339623634643433393135396264363835386135303061623964343832323936313536623634
-34396531643839623932663565656535396337353934373462336231356133303335643931323131
-33383030653362323830376435393134666339333036336333376539363033666666396534636131
-64353663663633313966336439643334616332616536336238613162363435323938313636303930
-66306135613735653437366164336237656331346365646265656362316639656162386239326632
-34613431653433306633666632363639306266303361383230316539306161396566346137363133
-30333131666530636264656263303639636161366134633964393834663436326539626264383034
-34636134663833333764613237663233346363643566353166316230396235373261393739316237
-38373466623635333432326531613637303839633033636166633364383964626134386261636536
-64616432343936363664383532323936386136393332653131343533653633323639356234353363
-62613631333864663435616633396537373035646238653565646535306163623262396136356663
-63383537653231633766613231316666303830336334323738626138366434323164643465663034
-64343965363738626638363733646136316338633161383330306363643461643664303734663863
-38656135616165366438623237333161363234356538356466646266316136313832666161626566
-65306234663338323263376437363961353033396532643132336261336662656136656530343532
-61353736626665386437376230633336656663356361626437393932633765336439613665666632
-34363763656164326565633433643836383234363962613535353230356461613133666232396336
-32316264353535353162373833666365633266376634366362646264616161363634363138303832
-30656465326562653430373962346438646138336364363833646165353636346161336336613565
-37303931393865356136333431656264666331313165373037656535333832643662343434396337
-37656263363161396438626437643839633834643265313963653337363463373530333930303163
-34353163333461353763643631303763623034333665373235656531393561363432633433393137
-63333438646565633533353466373930313831316132306662363831623732336435346334323331
-32373736396239613839396535316166386637613435643637323932303239643339613737653761
-63323062333430636161316437383035326531346466336263353761636661326363656631613063
-62623863616533333036643563333463366432386232386436356632323863663862613466613931
-35636538663263336333613664353934623338356262643832356163326161373166633833333838
-36373537363934393137663462633833373364356230356136646639633737636130396231633730
-61356536373064356665646335633435646164623164333964323135343536663162656564653961
-63626432653439373938333039386661363330373136633831656230336237376436313664653633
-61623830326330663235376534313135303566636131373835383534363730616433363536346666
-38326532376338393462363430643334343137613363626132356133363534643336663063623136
-66623261343932373662663134323538666334623764376131623530373065316338653535313165
-35313337303563373737313963303134303435643563616561323963613565646466353162353637
-62623961626630323432323133666636356531316362633331313630393761303030363736643163
-63333565616232366665356535373632626366366532633036323839393266613730373331343765
-39343132643734623162386162353438353165653537303033313439326132343036333739623832
-30316132396532626236633665333162653536633134313339306661613066343937323032376235
-64386161623639636361313865613536373338333931623438366338616338326536383232363564
-30356139373865313432366337646366613361393262333336643535656438383562343763636236
-38616263613834613030363033613839613334323964343436326136626565643838323064313037
-34663964646636363038393134343634663637663030636339613864316165666265383632373637
-32373063616130343762393832663030636334646263626236663730336135376461643134646337
-63346534353264663031336164623330633737333766633662613863343431333138363337326536
-38363937313632633537363261633133636266376534346430643230623663333464393262396564
-37323632653036653336623636646537326636383865383338366138653463336638363965356536
-30373664313863396237613363373236353966303862376561653562666231353930653233383733
-38303261343436393261653436626136353237666633613166623035636466346539363436616364
-36386438666165316137303932346435313736616434643337303038383166386135326132656265
-64633866363532333465623135323761393837623934316561633637613433316334376138343764
-64646262356631373865393363353465643733623535626434666639613965383239343736356231
-31306135666539646234333934326664353137643261613131646364363739333035396437316362
-62343531656234376438376266373932376233643630383836323032633537383961383537376631
-38303062623032636264666534313266363835636132646664366162303430393161313964616633
-32613063393965653438643766373434613237393736643635653465366533363535623932386433
-38653530316364323033383330373433663865393837653466376666633434343062376432633138
-64653365373062643431353533363764666339663334393432353466623062646632383163656365
-63633365326565353730333164303237383766386135376630613234333834626335643738376339
-31626161346562613439396263646361343464636136636663363766356263636630323366326131
-37356166343031376635346433363335373137303933343539303832613363656334663964333462
-31613531383166666261336439356434323263306364616536636333316131373930646262386366
-33333032643464313165336265326130333637323461643131393437326535316164356665313636
-61613237356331393463336431316539633133326334396134663035396531633538666331393132
-64393134643832376230656332633633396333313464333066656461366335343131613038656535
-32353037316132393936373834633535646631653464616331633830326131333361316464393762
-34306136306633323766353235353663666539663036383636393335336263353134303634626131
-31316334663733666530653330333263653135643539353439306165353339623032643633393632
-37653636663164646438626638643630393135623934646166666635316532366264623165616131
-64653736306431623639303564306431616263613966656563373861616339316466383761643439
-64323836303162613931313534393436353464333330353635663034396631636563616464653530
-34616164323362653231343433323338633930323035396630326461666461346338363039383737
-38666431373633376536323739316132316231353738306632336537633938653535316530346634
-65323937336635373736346438323133353333343235623463663264623838316239366165306535
-30323136626561636537393066383539303233366262396538383563353730356537623266623038
-33316135303833323663363733353634653763383166633663313465303838646266303264363339
-30363637396161656332323333653931336661393566663232326466646563356436306332333532
-35656331383661376538633664346537613161303135643937643161383034643532653961346533
-31396237653234363832383437613433326165636262333236393334613437386638636535616634
-34313330643139386232396332376135376332613333633938366533616435303232376366623237
-37323535383932623631363431383165323338306338633066373937323361383336616265303234
-32343132303366343866303837326566383865393764343130616366373536623766316238363430
-31633734633438313065336431316161636465333166653765653962326636663065336532393731
-33386139383466626565663666373664313030663132313233356136613435663265396332656633
-33636436646565643838643930343938326231623738393661393738343062303661353362646465
-31653335343638613238333636326337396438373263663666373263616662653634613036663364
-66323639336233623538663061366462663766383539333432336363613938346263636436336430
-36646639353163316661356636393034653934666563356335396631323166313532353361653630
-61353261656431613135313965373061313038653237626231366332306432633538656462323739
-38313831663861663738336564333731363937653663663031343261633538643535336264306230
-31656365326261633730613765393861376662656137623832363832633266346135643133633331
-39623836633337323934653464333731626361316638333565393435656132626631613739366535
-37653266333432386334333461653231346630646435343230636531326338396566396466376532
-32666632383430393064353861373033393538366266333339373436353932656333366264393936
-39373938346633616534643734313934643438333965323133313037373737303964643337653135
-31623038356263643438343635643530663362656630393164663634366461623464353532353239
-38623662666235383139663162343836316331396266313062316639373034376337393563643739
-65313564303962353061616265393135326266386266316538616430383966306163386533376137
-36393565336435636535376639623931616435353532356438666663383961356638353333366139
-63313962643638323862353738316233613862346532306631386536323261636235343565353563
-64613465343662663034643138623865363862316131363461343636333864656132626166393836
-32393737386166343630323861376562353337663030613263613664663938376337333533613432
-33343064636365643831373135623466613738303164326562386430386339653831313536366532
-32613430313461663662313966373238643934633938633764363365386139303337643533313961
-61646131653638616365346236616330363566636261636362363635323566353562326239393530
-33323934623530306436623963663735353337306162646638383934326530333163396165366139
-61653339363166376266643733653536636639653939346666663834376361373533363231636433
-35363665373031393231363161333036303239313032326134363432363034623762366637643830
-32353132353961343830303562613964376234303461623263343735336238353565353238363438
-61326164323265386431353130346364393466353334386336376139303036646634376331313561
-39386331366337646634366635623066313432623537323164646263323532303465343735326364
-33383162626139383232636362613831303832633562636136393138316334303832383666313963
-37643835663065613236303338353435663830343637326563343665616532633830623035336534
-39376536663937653939326261346263613265636363333531323931646633313837386238333435
-33646339626462386635316230636135626330643630633664636364313539626332643331356334
-32313736343031633631393662363638626137303331383263656335373966383335653133333932
-34393861303665613633366461666664376466383437303065653938323166633562643834356139
-32616564373731313134316661343839333830663338636132323265376133656337393632353931
-35633531383438656461613135323134623134393062356661326435323065616331656638306432
-62313130643763626336383762306535656238393961653537336465373064653432653665373362
-63666366343866613235376535313835623261313531653265373636383030613930353530346265
-64303531316239386630626565323966616535373936393766663063623464663864363338343466
-33666237613532326661626461343166323162366532386638333835663838343865346266646533
-66343533326562396331363137353631633965623561333963613565636264653031616339613232
-39666362643739663837326432623462646233306239326463616232383236346236383132396637
-34623332346533366362613465363263373339643033663264363332336463343337333466393866
-33333363313331376634363431366636376466336131363330323530306162666434393764393563
-35633530353839386362326262343635383434333338626563383463653263333430383465626535
-63363933343365326665343166633939313739316231383938376166646331623663343161623865
-36646630336337373865
+30353730633631376539373334376531353263396339313464373538396231616166653738383161
+6438656132616365646463653464396236313437323537620a363632643061333036353537626461
+34393065613836343536383565303066326133363937386263653961663666643430663139653961
+6462363235343063350a636165376232366636363233376637636231363839343738346134623562
+63663839326435303964313730336436396365343163343366613966323938656263303166326435
+65303931393233623064343763633161353161643635396365316362333137353732346366643162
+36386132386638623333623839373161386330353635346331393735366662393935353264646439
+36386539393432373765396562613033656339646532323864366230393638313161383363636636
+32353266376563393239393734316461313566386233333164623135303132346164353165323338
+62306333363838393236346435303339393639623231353232643966613163313134336265633636
+64366539326639303362393933303465376532626663346431613563353032383037363532646637
+38376266323537343437643635343634636164393230313632663533666434306134366261383230
+66383135303565383763646331383338646339613539623835366438393635363832366134316531
+63636466653239383231616139663231623636376564376234383762383066613030313937616533
+33396135616236643034303236353230376434313365326633303961303232363766633161353434
+64333832623535343733353462313862343934633363616237353331386264386632643538333939
+34636437393934326131363236346636663665613537343134313636623439356665636663613938
+37346634356164343537396363663730306232623964636635333333653738383538343236383535
+66326462373964333234343435336630643930333433393034303439366631386666393330343238
+61336237373236313533333732343064336637326636663433396362346631343165656163313561
+30383331326231383539343639343130323731333432306630353835343338613861376631646264
+66383539616663376237663234313431313064663566313566366664386439316161353135306338
+35373962636537653535333935306564303235356232313463386237353737313736386463396164
+62666165336539316565623634326562356434313833643833353635303361353937666663356263
+35653665323736386263333437666432623332333936373938376637633463653930326466383139
+38336239646135316565356564373731386662383234386433633632393165343331353835623437
+30663138633430356334353832633434633939373337663437326263336537356633313763353337
+64373764666430643433
@@ -1,44 +0,0 @@
---
-# Configure Audiobookshelf permissions
-# Fixes upload issues by ensuring audiobookshelf user can write to storage directories
-
- name: Configure Audiobookshelf
-  hosts: audiobookshelf.srv.renner.uno
-  become: true
-
-  tasks:
-    - name: Ensure audiobookshelf user exists
-      ansible.builtin.user:
-        name: audiobookshelf
-        system: true
-        create_home: false
-        state: present
-
-    - name: Set ownership on audiobooks directory
-      ansible.builtin.file:
-        path: /audiobooks
-        owner: audiobookshelf
-        group: audiobookshelf
-        mode: "0755"
-        state: directory
-        recurse: false
-
-    - name: Set ownership on audiobooks content recursively
-      ansible.builtin.command:
-        cmd: chown -R audiobookshelf:audiobookshelf /audiobooks
-      changed_when: true
-
-    - name: Set ownership on metadata directory
-      ansible.builtin.file:
-        path: /metadata
-        owner: audiobookshelf
-        group: audiobookshelf
-        mode: "0755"
-        state: directory
-        recurse: true
-
-    - name: Restart audiobookshelf service
-      ansible.builtin.systemd:
-        name: audiobookshelf
-        state: restarted
-        enabled: true
@@ -1,62 +0,0 @@
---
-# Setup PostgreSQL monitoring user and exporter
-# Usage: ansible-playbook -i inventory/ plays/setup-postgres-monitoring.yml
-
- name: Create PostgreSQL monitoring user on primary
-  hosts: authentik-db-1.srv.renner.uno
-  become: yes
-  vars_files:
-    - ../inventory/group_vars/all/vault.yml
-  tasks:
-    - name: Install psycopg2 for Ansible PostgreSQL modules
-      apt:
-        name: python3-psycopg2
-        state: present
-
-    - name: Create postgres_exporter user
-      become_user: postgres
-      community.postgresql.postgresql_user:
-        name: postgres_exporter
-        password: "{{ vault_postgres_exporter_password }}"
-        role_attr_flags: "NOSUPERUSER,NOCREATEDB,NOCREATEROLE"
-        state: present
-
-    - name: Grant pg_monitor role to exporter user
-      become_user: postgres
-      community.postgresql.postgresql_privs:
-        database: postgres
-        type: group
-        objs: pg_monitor
-        roles: postgres_exporter
-        grant_option: no
-
-    - name: Grant connect on authentik database
-      become_user: postgres
-      community.postgresql.postgresql_privs:
-        database: authentik
-        type: database
-        privs: CONNECT
-        roles: postgres_exporter
-
- name: Deploy postgres-exporter to Prometheus server
-  hosts: prometheus.srv.renner.uno
-  become: yes
-  vars_files:
-    - ../inventory/group_vars/all/vault.yml
-  vars:
-    postgres_exporter_db_host: "10.2.40.135"  # PostgreSQL HA VIP
-    postgres_exporter_db_user: "postgres_exporter"
-    postgres_exporter_db_password: "{{ vault_postgres_exporter_password }}"
-    postgres_exporter_db_name: "postgres"
-  roles:
-    - postgres-exporter
-
-  post_tasks:
-    - name: Verify postgres-exporter is running
-      uri:
-        url: "http://127.0.0.1:9187/metrics"
-        status_code: 200
-      register: exporter_health
-      retries: 3
-      delay: 5
-      until: exporter_health.status == 200
@@ -15,7 +15,14 @@
 #
 #   ansible-vault edit inventory/group_vars/all/vault.yml
 #   # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
-#   #        vault_netbird_proxy_token
+#   #        and netbird_turn_secret
+#
+# That last one is easy to miss and breaks the rotation silently. vault.yml
+# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
+# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
+# a group var outranks a role default — so the bare key wins, and rotating only
+# the vault_-prefixed one changes nothing that reaches the template. The check
+# below matches both spellings for that reason.
 #
 # The first rotation then populates netbird-rotatable.yml with fresh values, so
 # nothing needs copying across.
@@ -29,10 +36,24 @@
 #                               cannot disagree across a run.
 #   vault_netbird_relay_secret  management.json on the server and relay.env on
 #                               the proxies. Two hosts, both managed here.
-#   vault_netbird_proxy_token   proxy.env on the server and the proxies.
+#
+# The test for "safe to rotate here" is whether the value appears in
+# management.json.j2. If the management server does not read it from a file
+# Ansible writes, Ansible does not own both ends, whatever else it templates.
 #
 # WHAT IS NOT, and must not be added:
 #
+#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on both
+#                                the server and the proxies, all Ansible
+#                                templates — but the management server does not
+#                                validate it from any of them. It is registered
+#                                in the management DATABASE. Rotating it changes
+#                                only what the proxies present, so every proxy
+#                                is locked out: "proxy auth failed: too many
+#                                failed authentication attempts", and the
+#                                proxies stop receiving their domain
+#                                configuration. Learned by doing it.
+#
 #   vault_netbird_datastore_key  encrypts NetBird's datastore at rest. Changing
 #                                it without re-encrypting the store makes the
 #                                existing data unreadable. Rotating this needs a
@@ -114,7 +135,7 @@
          ansible-vault view
          --vault-password-file {{ playbook_dir }}/../vault-password.sh
          {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
-          | grep -cE '^vault_netbird_(turn_secret|relay_secret|proxy_token):'
+          | grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
      register: _stale_keys
      changed_when: false
      # grep exits 1 on "no matches", which is the good case here.
@@ -130,8 +151,8 @@
          rotates. group_vars files merge alphabetically and vault.yml sorts
          last, so it would override netbird-rotatable.yml on the next deploy
          and revert the secrets. Remove vault_netbird_turn_secret,
-          vault_netbird_relay_secret and vault_netbird_proxy_token from
-          vault.yml first (one-time migration), then re-run.
+          netbird_turn_secret and vault_netbird_relay_secret from vault.yml
+          first (one-time migration), then re-run.
        success_msg: vault.yml does not shadow the rotated keys.

  tasks:
@@ -150,7 +171,6 @@
      ansible.builtin.set_fact:
        new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
        new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
-        new_proxy_token: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
      no_log: true

    - name: Render the new values to a temporary file
@@ -163,7 +183,6 @@
          # the next rotation overwrites this file wholesale.
          vault_netbird_turn_secret: "{{ new_turn_secret }}"
          vault_netbird_relay_secret: "{{ new_relay_secret }}"
-          vault_netbird_proxy_token: "{{ new_proxy_token }}"
      no_log: true

    # Encrypting the whole file rather than inline !vault strings: these three
@@ -198,7 +217,6 @@
  vars:
    netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-    netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"

  roles:
    - netbird-server
@@ -213,7 +231,6 @@

  vars:
    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-    netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"

  roles:
    - netbird-proxy
@@ -227,7 +244,7 @@
    - name: Remaining manual steps
      ansible.builtin.debug:
        msg:
-          - "Rotated: turn_secret, relay_secret, proxy_token."
+          - "Rotated: turn_secret and relay_secret."
          - "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
          - "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
          - "OpenBao is unaffected by design — its NetBird credentials are separate ones."
@@ -1,19 +0,0 @@
---
-# Postgres Exporter defaults
-
-postgres_exporter_version: "0.15.0"
-postgres_exporter_port: 9187
-postgres_exporter_user: postgres-exporter
-postgres_exporter_group: postgres-exporter
-
-# PostgreSQL connection (use vault for password)
-postgres_exporter_db_host: "localhost"
-postgres_exporter_db_port: 5432
-postgres_exporter_db_user: "postgres_exporter"
-postgres_exporter_db_password: "{{ vault_postgres_exporter_password | default('changeme') }}"
-postgres_exporter_db_name: "postgres"
-postgres_exporter_db_sslmode: "disable"
-
-# Installation paths
-postgres_exporter_install_dir: /opt/postgres-exporter
-postgres_exporter_binary: "{{ postgres_exporter_install_dir }}/postgres_exporter"
@@ -1,9 +0,0 @@
---
- name: Reload systemd
-  systemd:
-    daemon_reload: yes
-
- name: Restart postgres-exporter
-  systemd:
-    name: postgres-exporter
-    state: restarted
@@ -1,97 +0,0 @@
---
-# Install and configure Postgres Exporter
-
- name: Create postgres-exporter system group
-  group:
-    name: "{{ postgres_exporter_group }}"
-    system: yes
-    state: present
-
- name: Create postgres-exporter system user
-  user:
-    name: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    home: "{{ postgres_exporter_install_dir }}"
-    createhome: no
-
- name: Create installation directory
-  file:
-    path: "{{ postgres_exporter_install_dir }}"
-    state: directory
-    owner: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    mode: '0755'
-
- name: Check if postgres_exporter is already installed
-  stat:
-    path: "{{ postgres_exporter_binary }}"
-  register: postgres_exporter_installed
-
- name: Get installed version
-  command: "{{ postgres_exporter_binary }} --version"
-  register: installed_version
-  when: postgres_exporter_installed.stat.exists
-  changed_when: false
-  failed_when: false
-
- name: Download postgres_exporter
-  get_url:
-    url: "https://github.com/prometheus-community/postgres_exporter/releases/download/v{{ postgres_exporter_version }}/postgres_exporter-{{ postgres_exporter_version }}.linux-amd64.tar.gz"
-    dest: /tmp/postgres_exporter.tar.gz
-    mode: '0644'
-  when: not postgres_exporter_installed.stat.exists or (installed_version.stdout is defined and postgres_exporter_version not in installed_version.stdout)
-
- name: Extract postgres_exporter
-  unarchive:
-    src: /tmp/postgres_exporter.tar.gz
-    dest: /tmp
-    remote_src: yes
-  when: not postgres_exporter_installed.stat.exists or (installed_version.stdout is defined and postgres_exporter_version not in installed_version.stdout)
-
- name: Install postgres_exporter binary
-  copy:
-    src: "/tmp/postgres_exporter-{{ postgres_exporter_version }}.linux-amd64/postgres_exporter"
-    dest: "{{ postgres_exporter_binary }}"
-    owner: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    mode: '0755'
-    remote_src: yes
-  when: not postgres_exporter_installed.stat.exists or (installed_version.stdout is defined and postgres_exporter_version not in installed_version.stdout)
-  notify: Restart postgres-exporter
-
- name: Create postgres_exporter environment file
-  template:
-    src: postgres_exporter.env.j2
-    dest: "{{ postgres_exporter_install_dir }}/postgres_exporter.env"
-    owner: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    mode: '0600'
-  notify: Restart postgres-exporter
-
- name: Create systemd service
-  template:
-    src: postgres_exporter.service.j2
-    dest: /etc/systemd/system/postgres-exporter.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart postgres-exporter
-
- name: Enable and start postgres-exporter service
-  systemd:
-    name: postgres-exporter
-    enabled: yes
-    state: started
-    daemon_reload: yes
-
- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - /tmp/postgres_exporter.tar.gz
-    - "/tmp/postgres_exporter-{{ postgres_exporter_version }}.linux-amd64"
@@ -1,2 +0,0 @@
-# PostgreSQL connection string for postgres_exporter
-DATA_SOURCE_NAME="postgresql://{{ postgres_exporter_db_user }}:{{ postgres_exporter_db_password }}@{{ postgres_exporter_db_host }}:{{ postgres_exporter_db_port }}/{{ postgres_exporter_db_name }}?sslmode={{ postgres_exporter_db_sslmode }}"
@@ -1,23 +0,0 @@
-[Unit]
-Description=Prometheus PostgreSQL Exporter
-Documentation=https://github.com/prometheus-community/postgres_exporter
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ postgres_exporter_user }}
-Group={{ postgres_exporter_group }}
-EnvironmentFile={{ postgres_exporter_install_dir }}/postgres_exporter.env
-ExecStart={{ postgres_exporter_binary }} \
-  --web.listen-address=:{{ postgres_exporter_port }} \
-  --collector.stat_statements
-Restart=on-failure
-RestartSec=10s
-
-# Security hardening
-NoNewPrivileges=yes
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target
@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```

 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).
@@ -0,0 +1,194 @@
+# Proposal — separate the four traffic classes onto their own VLANs
+
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
+
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
+
+## The four classes
+
+| class | what it is | where it goes today |
+|---|---|---|
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
+
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
+
+## Principle
+
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
+
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
+
+## The finding that shapes everything else
+
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
+
+`installDirectRoute` takes the next-hop as a parameter:
+
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
+|---|---|---|
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
+
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
+general.
+
+## Recommended mapping
+
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
+
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
+
+Under B:
+
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
+
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.
@@ -31,8 +31,8 @@ all in state that outlived the VM.
      two hosts every PG is undersized with min_size 2, so one further OSD loss
      blocks IO on every CephFS mount cluster-wide.
 - [ ] Talos metal ISO on a USB stick. Schematic
-      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
-      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
+      `0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9`
+      (`net.ifnames=1`, intel-ucode, thunderbolt, i915; no qemu-guest-agent).
      Verify by comparing partition offsets and sizes against the ISO with
      `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is
@@ -383,6 +383,34 @@ raised to `high_recovery_ops` to speed the evacuation.
 The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
 `ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.

+### Long-lived TCP sessions to devices outside the cluster die silently
+
+Check this before declaring the conversion done. Anything holding a persistent
+connection to a device on another VLAN — the ESPHome integration is the case
+that bit us — keeps a half-open socket when the node underneath it moves. The
+peer drops its side; the pod's side never learns, because no RST arrives and no
+keepalive fires.
+
+The symptom is that there is no symptom. Home Assistant logged nothing, retried
+nothing, and reported the devices as connected. The devices themselves showed
+"not connected". Hours went into reading logs that were empty by construction.
+
+`/proc/net/tcp` in the pod settles it in one command — `01` is ESTABLISHED,
+`02` is SYN_SENT, and the port is hex:
+
+```sh
+kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \
+  sh -c 'grep ":17A5" /proc/net/tcp'      # 17A5 = 6053, the ESPHome API
+```
+
+Seven `01` entries to devices insisting they were disconnected is the tell. A
+`02` alongside them is a genuinely dead device, not a stale socket — that one
+Home Assistant *was* retrying and *did* log.
+
+Restarting the consuming pod clears it. Reloading just the integration from the
+UI does the same with less disruption. Neither is discoverable from logs, which
+is why it belongs here.
+
 ## Known-stale things that will bite

 - `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
@@ -1,10 +1,25 @@
 # Cold-rebuild runbook — homelab cluster from scratch

 Rebuild the entire cluster from **this repo + backups only**, assuming nothing
-in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
+in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
+gone).

 Order: **tofu → talos → flux → openbao → apps.**

+All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
+are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
+Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
+resources, and physically booting three machines from install media.
+
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.
+
 ## The circular-bootstrap problem this runbook exists to break

 In steady state Flux's `GitRepository flux-system` pulls manifests from the
@@ -61,12 +76,21 @@ These must exist outside the cluster or the rebuild cannot complete:
      `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
      copy is the one that survives.
 - [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
-      OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
+      OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
+      Proxmox credentials — there is no Proxmox any more.
+- [ ] **Install media** written and to hand: the Image Factory ISO named in
+      Step 1, on a USB stick, for all three machines.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.

-## Step 1 — tofu: provision the VMs + supporting infra
+## Step 1a — tofu: DNS, object storage, UniFi

-Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
-UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
+OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
+the tree but nothing instantiates it, so `tofu apply` here creates only the
+platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
+records, the Hetzner object storage, and the NAT hairpin rules.

 ```sh
 cd opentofu
@@ -74,44 +98,119 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
 tofu apply
 ```

-In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
-which also pre-seeds the Talos image onto every Proxmox node. Wait until the
-node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
-Step 2 — `bootstrap.sh` polls DNS but will not create the records.
+Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
+FQDNs to resolve — the control-plane endpoint is baked into every generated
+config, and the nodes are addressed by name.

-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
+## Step 1b — boot all three machines from install media

 ```
-https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
 ```

-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
+Write it to a USB stick and boot each MS-01 from it. Each comes up in
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.

- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
+The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
+interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
+against do not exist, and the node ends up with no address on any interface.
+Beyond that it carries:
+
+- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
+  today; present so wanting hardware transcoding later is a config change
+  rather than another rolling reinstall
+- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
+  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
+  rule auto-authorises any Thunderbolt device attached once those modules load
+- no `qemu-guest-agent`, which has nothing to talk to on bare metal
+
+Verify the disks before going further. Talos enumerates NVMe in an order that
+is not stable across boots, and the per-node configs select by serial for
+exactly that reason — the failure mode is the installer taking the Ceph OSD
+drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
+NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
+in `talos/patches/k8s-N.yaml`.

 ## Step 2 — talos: form the cluster + get kubeconfig

+**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
+config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
+`$patch: delete` document fails against a base with no `inlineManifests`
+(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
+and 6; only config generation is broken.
+
+The procedure below is what actually works, and is the same one
+`RUNBOOK-baremetal-conversion.md` uses.
+
+### Generate the configs
+
+This is a genuinely new cluster — nothing survives to copy secrets from, so
+generate them fresh. (In a *partial* rebuild, where one node still runs, pull
+secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
+--from-controlplane-config`, or the new node forms its own cluster rather than
+joining.)
+
 ```sh
-kubernetes/bootstrap.sh talos
+talosctl gen secrets -o secrets.yaml
+
+for N in 1 2 3; do
+  talosctl gen config homelab https://k8s.renner.uno:6443 \
+    --with-secrets secrets.yaml --output-types controlplane \
+    --config-patch @talos/patches/cluster.yaml \
+    --config-patch @<cilium additive half> \
+    --config-patch @talos/patches/registry-mirror.yaml \
+    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
+done
 ```

-Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
-writes `~/.talos/config` + `~/.kube/config`, and stages both into
-`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
+Two things the generator gets wrong for this repo, both of which will bite
+silently:
+
+- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
+  second document. See issue #581.
+- **Strip the generated `HostnameConfig`** — it collides with
+  `machine.network.hostname` from the per-node patch.
+
+Validate each before applying:
+
+```sh
+talosctl validate --config install-k8s-$N.yaml --mode metal
+```
+
+### Apply in two stages
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
+and nothing times out. Concretely, the `ens19` VLAN document must be removed
+from the first-stage config — its parent exists, but the install stalls anyway.
+
+```sh
+# Stage 1: minimal config, from the maintenance-mode address off the console
+talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
+# Stage 2: once the node is up on management, apply the complete config
+talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
+```
+
+`--mode=try` rolls back on its own if the node becomes unreachable, which is the
+only safety net there is for a network change that can strand a machine with no
+IPMI. Use it for every network-affecting apply, then make it permanent once
+proven.
+
+### Bootstrap etcd, then collect access
+
+Bootstrap **exactly one** node:
+
+```sh
+talosctl -n k8s-1.srv.renner.uno bootstrap
+talosctl -n k8s-1.srv.renner.uno kubeconfig .
+```
+
+Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
+supported way to get the kubeconfig into place afterwards.

 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)

@@ -149,8 +248,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).

 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods

 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +257,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap

-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```

 Verify the store recovers before moving on:
@@ -170,7 +273,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```

 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.

 ## Step 5 — apps: let the rest reconcile

@@ -225,9 +330,28 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
  in-cluster commits made **after** Flux/Gitea broke are not on the mirror
  (Gitea couldn't push them if Gitea itself was down). This is the intended
  recovery point, but be aware of it when reconciling drift.
- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
-  key must not live in CI. See issue #187 for the open question of whether to
-  wire a semi-automated OpenBao restore.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
+- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
+  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
+  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
+  It has also never had an `openbao` phase — Step 4 is manual because the
+  unseal key must not live in CI; see issue #187 on whether to wire a
+  semi-automated OpenBao restore.
+- **OpenBao seals on every restart and each replica must be unsealed
+  individually.** Three raft replicas mean the service survives losing one, but
+  a sealed server is neither a voter nor a standby — unsealing only one leaves
+  the cluster short of quorum. Step 4 covers this; it is repeated here because
+  it is the step most likely to be half-done under pressure.
+- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
+  IPAM, not host networking. Nothing in the rebuild needs special handling for
+  it — the nodes hold no address on that segment — but a rebuilt cluster will
+  not have the whereabouts allocations from before, which is harmless since
+  they are assigned fresh.
 - warn-and-continue waits in the `flux` phase can mask a genuinely broken
  Ceph/infra bring-up; always read the phase output rather than trusting the
  exit code.
@@ -37,6 +37,13 @@ resources:
  - esphome-service.yaml
  - esphome-gate-service.yaml
  - esphome-ingress.yaml
+  # mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
+  # unicast, which is the only direction the CNI will move them. Without it HA
+  # never learns that a device's DHCP lease moved. See the header comment in
+  # mdns-relay-deployment.yaml.
+  - mdns-relay-network-attachment.yaml
+  - mdns-relay-service.yaml
+  - mdns-relay-deployment.yaml
  # Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
  # runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
  # `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
@@ -0,0 +1,190 @@
+---
+# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
+# without putting a macvlan on the HA pod itself.
+#
+# THE PROBLEM
+#
+# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
+# ff02::fb — and never hears anything on either.
+# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
+# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
+# it does not replicate multicast between pods either. Both halves of the path
+# are dead, which is why an avahi-style reflector — receive on one interface,
+# re-emit as multicast on the other — cannot work here: the re-emitted packets
+# would be dropped by the CNI.
+#
+# Consequence for HA: the ESPHome integration dials devices at the address
+# cached in its config entry. When a lease rotates, every attempt fails with
+# `[Errno 113] Connect call failed` and the device stays unreachable, because
+# the mDNS announcement that would carry the new address never arrives. New
+# devices are never discovered at all.
+#
+# THE MECHANISM
+#
+# This relay holds the multicast membership on the IoT VLAN through its own
+# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
+# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
+# the CNI will carry.
+#
+# It works because python-zeroconf does not filter on how a packet arrived:
+# its datagram handler validates size, deduplicates, and parses, but performs
+# no check on destination address, source address, or source port. A unicast
+# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
+# unicast responses are part of RFC 6762 — the QU bit — so this is within the
+# protocol, not a trick.)
+#
+# WHAT THIS DOES AND DOES NOT RESTORE
+#
+# Restored: passive reception. RFC 6762 requires a responder to re-announce
+# when its address changes, so a device that moves to a new lease announces
+# the move, HA sees it, and the ESPHome integration updates the config entry.
+# That is the failure this exists to fix. New devices announcing themselves
+# are likewise seen.
+#
+# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
+# multicast and die there, so HA cannot actively solicit a response. Anything
+# depending on a query rather than an announcement stays broken. Making that
+# work needs a querier on the VLAN side, which this relay deliberately is not.
+#
+# EXPOSURE
+#
+# The receive sockets carry so-bindtodevice=net1, so only datagrams that
+# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
+# address instead is not possible — it is DHCP-leased and unknown at exec time
+# — but binding to the device achieves the same confinement without needing to
+# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
+# unprivileged uid here.
+#
+# Without it socat would bind the wildcard, and since the namespace CNP grants
+# every pod in home-assistant unrestricted access to every other pod on every
+# port, any sibling pod could reach :5353 over the cluster network and have
+# records forwarded into HA's zeroconf cache.
+#
+# What remains, and cannot be closed here: any host on the IoT VLAN can send
+# this port a datagram. That is the same trust already extended to genuine
+# mDNS from that segment, which any device there can equally forge — the relay
+# widens who can reach HA's listener, not what can be claimed. macvlan traffic
+# bypasses Cilium policy entirely, so the namespace CNP does not constrain
+# this interface either way.
+#
+# Note that the relay is not what makes HA's zeroconf reachable from inside
+# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
+# and the namespace CNP already permits any sibling pod to reach it directly.
+# Confining the relay removes a second route to that listener; it does not
+# remove the first.
+#
+# Nothing else is exposed: the containers run socat and only socat, with no
+# second listener to reach.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mdns-relay
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: mdns-relay
+spec:
+  # One replica. Extra replicas would deliver a duplicate of every announcement
+  # — harmless, since zeroconf deduplicates, but they buy nothing: this is a
+  # cache-refresh path, and a missed window is corrected by the next
+  # announcement.
+  replicas: 1
+  # Recreate, not RollingUpdate: two relays would briefly hold two DHCP leases
+  # on v30iot for one workload.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mdns-relay
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mdns-relay
+      annotations:
+        k8s.v1.cni.cncf.io/networks: mdns-relay-iot
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      # One container per address family. mDNS is two protocols — 224.0.0.251
+      # and ff02::fb — and a socket joined to one hears nothing of the other.
+      # HA listens on both (its zeroconf holds both groups on eth0), so a
+      # v4-only relay would silently drop every IPv6 announcement, and devices
+      # that announce only over v6 would stay invisible exactly as before.
+      #
+      # Two processes rather than one: socat carries a single address pair per
+      # invocation, and the families need different receive and send addresses.
+      # They are independent — one crashing does not blind the other — and each
+      # is restarted by the kubelet on its own.
+      containers:
+        # -u is load-bearing on both: unidirectional, left to right. Without it
+        # socat would also carry HA's replies back out, and this must stay a
+        # one-way tap.
+        #
+        # UDP{4,6}-RECV (not -RECVFROM) keeps reading from one socket instead of
+        # forking per datagram — the IoT VLAN's mDNS chatter is continuous, and
+        # a fork per packet would be a process storm.
+        #
+        # The group is joined on net1 specifically, so the relay subscribes on
+        # the IoT VLAN rather than on the pod network.
+        #
+        # Both send targets are the same ClusterIP name, resolved once at start;
+        # UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
+        # mdns-relay-service.yaml for why that indirection is required and why
+        # the Service is dual-stack.
+        - name: socat-ipv4
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
+            - "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          # No probes. socat is a single process with no health surface: if it
+          # exits, the container exits and the kubelet restarts it, which is the
+          # entire liveness story. A probe here could only re-assert that PID 1
+          # is running.
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+        # Joining ff02::fb needs no address configuration on net1: the group is
+        # link-local and the kernel's automatic link-local address on the
+        # macvlan is sufficient to receive on it. Whether v30iot hands out
+        # global v6 addresses does not affect reception here.
+        #
+        # ipv6-v6only=1 keeps this socket to v6 only. Both containers share the
+        # pod's network namespace, and a dual-stack v6 socket would overlap the
+        # v4 container's port. Measured behaviour without it was no contention —
+        # the v4 receiver kept taking packets while the v6 one ran — but that
+        # depends on the kernel's bindv6only default rather than on anything
+        # stated here, so the roles are made explicit instead of inherited.
+        - name: socat-ipv6
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
+            - "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
@@ -0,0 +1,52 @@
+---
+# mdns-relay's second NIC: macvlan onto v30iot, so the relay has real L2
+# presence on the IoT VLAN and can join both mDNS multicast groups
+# (224.0.0.251 and ff02::fb, port 5353). The v6 group needs no address
+# configuration of its own: it is link-local, and the kernel's automatic
+# link-local address on the macvlan is enough to receive on it, independently
+# of whether v30iot hands out global v6 addresses.
+# Multicast is link-local — it does not cross the L3
+# boundary between the cluster pod CIDR and v30iot — so that membership has to
+# be held by a pod sitting physically on that segment.
+#
+# The address is leased from the UniFi DHCP server on v30iot. Nothing
+# references it: the relay only receives on this interface and sends onward
+# over eth0. See mdns-relay-deployment.yaml for what the relay does with it.
+#
+# Multus's `master` is an interface name. On the k8s-{1,2,3} nodes the v30iot
+# NIC is `ens19` — a stable, busPath-derived predictable name (net.ifnames=1,
+# via the Image Factory schematic), identical across nodes and independent of
+# driver probe order. This is the same master shelly-manager-iot uses.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: mdns-relay-iot
+  namespace: home-assistant
+spec:
+  # Chained CNI, identical in shape to shelly-manager-iot: macvlan leases the
+  # v30iot address by DHCP, then `sbr` confines net1's routes to a
+  # source-based routing table. Without sbr the DHCP-supplied default gateway
+  # (10.2.30.1, option 3) installs a second default route at the same metric
+  # as eth0's, and cluster-internal traffic egresses the IoT VLAN and is
+  # dropped. The relay's forward path to Home Assistant runs over eth0, so
+  # that route has to stay intact.
+  #
+  # Requires the cni-dhcp-daemon DaemonSet in infrastructure/multus.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "mdns-relay-iot",
+      "plugins": [
+        {
+          "type": "macvlan",
+          "master": "ens19",
+          "mode": "bridge",
+          "ipam": {
+            "type": "dhcp"
+          }
+        },
+        {
+          "type": "sbr"
+        }
+      ]
+    }
@@ -0,0 +1,42 @@
+---
+# Stable delivery target for the relay's forwarded mDNS traffic.
+#
+# The relay resolves its send target once, at process start (socat has no
+# re-resolution loop), so it cannot chase the HA pod IP across restarts. A
+# ClusterIP is the fix: the IP is stable for the lifetime of the Service, and
+# Cilium re-DNATs to whatever pod currently backs it.
+#
+# Deliberately a separate Service from `home-assistant`, which targets the
+# nginx proxy sidecar on 8124 and is the XFF trust anchor for the public UI.
+# This one bypasses the sidecar and addresses the home-assistant container's
+# own port directly — containers in a pod share a network namespace, so
+# targetPort 5353 lands on python-zeroconf's listener regardless of which
+# container opened it. Keeping the two apart means the UI path and the
+# discovery path cannot be changed by accident through one another.
+#
+# No CiliumNetworkPolicy change is needed: home-assistant-ns.yaml already
+# permits intra-namespace traffic on all ports, and the relay is in this
+# namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: home-assistant-mdns
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: home-assistant
+spec:
+  # Dual-stack, because mDNS is two protocols and HA listens on both: its
+  # zeroconf socket holds 224.0.0.251 on eth0 and ff02::fb alongside it. The
+  # relay forwards each family over its own transport rather than folding v6
+  # announcements into a v4 hop, so the Service has to answer on both.
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: home-assistant
+  ports:
+    - name: mdns
+      port: 5353
+      protocol: UDP
+      targetPort: 5353
@@ -6,12 +6,14 @@ kind: Kustomization
 resources:
  - prometheusrule-ceph.yaml
  - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
  - prometheusrule-mariadb-backup.yaml
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
  - prometheusrule-velero.yaml
  - prometheusrule-woodpecker-rotation.yaml
  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
@@ -23,4 +25,5 @@ resources:
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
  - probes-blackbox.yaml
@@ -0,0 +1,121 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # THE GAP THESE RULES CLOSE
+        #
+        # Drive wear is invisible in every other signal this cluster has.
+        # `ceph -s` reads active+clean at any wear level, no Ceph health check
+        # fires until a device actually fails, and the block-layer metrics show
+        # throughput and latency but never endurance. That blind spot is how a
+        # set of OSD drives reached 81-87% before anyone looked (#549, #434).
+        #
+        # The numbers below are wear percentages reported by the drive itself:
+        # 100 means the manufacturer's rated endurance is used up. Drives do
+        # not stop at 100 — they typically keep working and start consuming
+        # spare blocks — so treat these as planning thresholds, not failure
+        # predictions.
+        - alert: NvmeDriveWearHigh
+          # 80% with months of runway left is the point where ordering a
+          # replacement is still a calm decision rather than an outage.
+          # `for` is long because wear is monotonic and slow: it cannot
+          # flap, and a short window would only make the alert noisier
+          # after a restart.
+          expr: smartctl_device_percentage_used >= 80
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive has used {{ $value }}% of its rated write endurance.
+              Nothing is failing and no data is at risk yet. Check what the
+              drive is carrying before deciding urgency: an OSD is replaceable
+              with a rebalance, whereas the EPHEMERAL volume has no such
+              escape — a node simply stops when it goes. The Disk Health
+              dashboard resolves which drive holds which role.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # A drive that has consumed its spare area is no longer masking media
+        # defects, which is a materially different state from being worn:
+        # wear is a budget, spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          # The NVMe critical warning byte is a bitfield: spare below
+          # threshold, temperature out of range, degraded reliability,
+          # read-only mode, volatile-memory backup failed. Any non-zero value
+          # is the drive itself asking for attention.
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # Unlike wear, this counter should never move. Alert on the increase
+          # rather than the absolute value, so a drive that recorded errors
+          # long ago does not alert forever while a drive that starts failing
+          # today does.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # Without this the whole group is silently useless: no series means no
+        # alert fires, and a dashboard of empty panels reads much like a
+        # healthy one. This is the check that the checks are running.
+        - alert: SmartctlExporterIncomplete
+          # Compared against node-exporter's own NVMe inventory rather than a
+          # written-down fleet size. Adding or removing a drive then moves both
+          # sides together, so this cannot rot into a permanent false alarm the
+          # way a hardcoded count would.
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.
@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.
@@ -40,17 +40,31 @@ spec:
            description: "A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."

        - alert: VeleroBackupStale
-          # Daily schedule with no successful backup in over 30h: covers
-          # the case where multiple consecutive runs failed (Garage flapping
-          # over the backup window) without the per-event alerts being
-          # acted on.
+          # Daily schedule with no successful backup in over 60h: covers the
+          # case where consecutive runs failed without the per-event alerts
+          # being acted on.
+          #
+          # 60h, not 30h. At 30h a single missed cycle pages, and a single
+          # missed cycle is the normal cost of touching the nodes: Velero moves
+          # data through pods, so draining a node kills whatever upload is in
+          # flight and the schedule is simply lost for that day — it is not
+          # retried. Rolling node work between 03:15 and 08:45 UTC therefore
+          # produced a guaranteed critical alert for every schedule it
+          # overlapped, which is noise about planned work rather than a signal
+          # about backups.
+          #
+          # 60h still catches what this is for. Two consecutive failures is no
+          # longer explicable by one maintenance window, and the per-event
+          # VeleroBackupFailed alert fires on the first failure regardless — so
+          # nothing is hidden, only the escalation to critical is deferred
+          # until a second cycle confirms it.
          expr: |
-            time() - velero_backup_last_successful_timestamp{schedule!=""} > 30 * 3600
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
            and
            velero_backup_last_successful_timestamp{schedule!=""} > 0
          for: 10m
          labels:
            severity: critical
          annotations:
-            summary: "Velero {{ $labels.schedule }} has no successful backup in 30h+"
-            description: "Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
+            summary: "Velero {{ $labels.schedule }} has no successful backup in 60h+"
+            description: "Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
@@ -0,0 +1,29 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
+    # by this label; without it the resource exists but is never
+    # discovered.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl itself polls the drives every 60s, so scraping faster than that
+    # returns the same numbers with a new timestamp. Wear moves on a scale of
+    # weeks; 60s is already far finer than the signal.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this every series is labelled with the exporter pod's IP and
+        # nothing else. A DaemonSet pod's IP says nothing about which machine
+        # the drive is in, and it changes on every reschedule — so the wear
+        # figure would be unattributable, which is most of its value gone.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node
@@ -0,0 +1,382 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"}
+      ],
+      "title": "OSDs down / out",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 40},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 48},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}
@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
  - name: dashboard-blackbox-overview
    files:
      - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
  - name: dashboard-gitea-overview
    files:
      - gitea-overview.json
@@ -275,8 +275,11 @@ spec:
          - alertmanager-smtp
      # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
      # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
      # monitoring ns is not blocked by any CNP today.
      config:
        global:
@@ -15,6 +15,7 @@ resources:
  - prometheus-gate-service.yaml
  - prometheus-gate-ingress.yaml
  - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
  - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -0,0 +1,158 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# WHY THIS HOLDS ELEVATED CAPABILITIES, and why that is a deliberate exception
+# rather than an oversight:
+#
+# NVMe reports health over an admin ioctl on the character device
+# (NVME_IOCTL_ADMIN_CMD), which the kernel gates on CAP_SYS_ADMIN. There is no
+# sysfs path for wear, media errors or written-bytes totals — hwmon publishes
+# the controller temperature and nothing else. Upstream smartctl_exporter
+# documents `privileged: true` and `user: root` for exactly this reason.
+#
+# The alternative that needs no privileges — reading Ceph's devicehealth
+# archive, which does hold full smartctl output — covers OSD drives only. In
+# this cluster that is three of nine, and the six it misses include the drives
+# that were retired from Ceph for wear and now carry EPHEMERAL. Covering the
+# healthy drives and not the worn ones was judged not worth a new service; the
+# owner chose fleet-wide coverage with the privilege exception instead.
+#
+# The exception is `privileged: true`, and narrower settings were tried and
+# measured to fail — see the securityContext below for what was ruled out and
+# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork,
+# read-only root filesystem, and /dev as the only host mount.
+#
+# If this ever needs revisiting, there are exactly two things worth checking,
+# and neither is "does another exporter ask for less":
+#   - whether the kernel has grown a sysfs interface for NVMe health, which
+#     would remove the need for device access altogether
+#   - whether enough workloads need raw device access to justify a device
+#     plugin, which is the only way Kubernetes grants individual devices
+# The requirement comes from the device cgroup and the ioctl gate, not from
+# the choice of tool.
+#
+# Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
+# The repository is not archived; the pin is simply its newest tag.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device on purpose: the exporter scans for devices
+          # itself and rescans every 10m by default. Device names are not
+          # stable across these machines, so a hardcoded list would quietly
+          # stop covering a drive the next time one is replaced.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            # PRIVILEGED, and there is no narrower setting that works. This
+            # was established by measurement after two narrower attempts
+            # failed, so the reasoning is recorded here rather than rediscovered:
+            #
+            #   Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else
+            #   dropped. The exporter found all three drives per node and
+            #   could open none: "Smartctl open device: /dev/nvme0 failed:
+            #   Operation not permitted".
+            #
+            #   Attempt 2 — the same plus seccompProfile: Unconfined, on the
+            #   theory that EPERM is the default seccomp action for an
+            #   unlisted syscall and the kubelet runs seccompDefault: true.
+            #   /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both
+            #   capabilities were effective and a filter was loaded. Removing
+            #   the filter changed nothing.
+            #
+            #   What actually settled it: `head -c 512 /dev/nvme0n1` inside
+            #   the container fails with EPERM too. A plain read of a block
+            #   device involves no ioctl, no capability and no syscall a
+            #   seccomp profile would filter. The block is the device cgroup —
+            #   containerd denies non-privileged containers access to host
+            #   device nodes whether or not /dev is mounted.
+            #
+            # Kubernetes exposes no per-device allowlist in a Pod spec; there
+            # is no equivalent of `docker --device`. The alternatives are
+            # `privileged: true` or a device plugin advertising the NVMe
+            # drives as a resource. A device plugin is the genuinely narrower
+            # answer and is worth building if more workloads ever need raw
+            # device access; for a single read-only metrics exporter it is a
+            # component to write, deploy and maintain in exchange for the same
+            # access.
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: true
+            # Stated rather than omitted: the API rejects `privileged: true`
+            # together with `allowPrivilegeEscalation: false`, so leaving it
+            # out would only hide that privileged already implies it.
+            allowPrivilegeEscalation: true
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            # Not read-only. smartmontools opens NVMe devices read-write on
+            # some paths, and once the container is privileged a read-only
+            # flag on this mount is not a boundary anyone should rely on —
+            # keeping it would suggest a protection that is not there.
+            - name: dev
+              mountPath: /dev
+            # smartctl is not known to need scratch space, but the root
+            # filesystem is read-only and a tool that turns out to want /tmp
+            # would crashloop instead of degrading. Cheaper to provide it than
+            # to find out from a restart counter.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP
@@ -55,7 +55,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -65,7 +65,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.
@@ -4,3 +4,12 @@ resources:
  - cni-plugins-installer.yaml
  - cni-dhcp-daemon.yaml
  - multus-daemonset.yaml
+  # Whereabouts — cluster-coordinated IPAM, for attachments on segments with
+  # no DHCP server. CRDs first: the DaemonSet's control loop reads them at
+  # startup. See the header in whereabouts-daemonset.yaml for why neither
+  # dhcp nor host-local covers the Ceph SAN.
+  - whereabouts-crd-ippools.yaml
+  - whereabouts-crd-overlappingrangeipreservations.yaml
+  - whereabouts-crd-nodeslicepools.yaml
+  - whereabouts-daemonset.yaml
+  - whereabouts-reconciler.yaml
@@ -0,0 +1,71 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: ippools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: IPPool is the Schema for the ippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPPoolSpec defines the desired state of IPPool
+            properties:
+              allocations:
+                additionalProperties:
+                  description: IPAllocation represents metadata about the pod/container
+                    owner of a specific IP
+                  properties:
+                    id:
+                      type: string
+                    ifname:
+                      type: string
+                    podref:
+                      type: string
+                  required:
+                  - id
+                  - podref
+                  type: object
+                description: |-
+                  Allocations is the set of allocated IPs for the given range. Its` indices are a direct mapping to the
+                  IP with the same index/offset for the pool's range.
+                type: object
+              range:
+                description: Range is a RFC 4632/4291-style string that represents
+                  an IP address and prefix length in CIDR notation
+                type: string
+            required:
+            - allocations
+            - range
+            type: object
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,79 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: nodeslicepools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: NodeSlicePool
+    listKind: NodeSlicePoolList
+    plural: nodeslicepools
+    singular: nodeslicepool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: NodeSlicePool is the Schema for the nodesliceippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: NodeSlicePoolSpec defines the desired state of NodeSlicePool
+            properties:
+              range:
+                description: |-
+                  Range is a RFC 4632/4291-style string that represents an IP address and prefix length in CIDR notation
+                  this refers to the entire range where the node is allocated a subset
+                type: string
+              sliceSize:
+                description: SliceSize is the size of subnets or slices of the range
+                  that each node will be assigned
+                type: string
+            required:
+            - range
+            - sliceSize
+            type: object
+          status:
+            description: NodeSlicePoolStatus defines the desired state of NodeSlicePool
+            properties:
+              allocations:
+                description: Allocations holds the allocations of nodes to slices
+                items:
+                  properties:
+                    nodeName:
+                      description: NodeName is the name of the node assigned to this
+                        slice, empty node name is an available slice for assignment
+                      type: string
+                    sliceRange:
+                      description: SliceRange is the subnet of this slice
+                      type: string
+                  required:
+                  - nodeName
+                  - sliceRange
+                  type: object
+                type: array
+            required:
+            - allocations
+            type: object
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,57 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: overlappingrangeipreservations.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: OverlappingRangeIPReservation
+    listKind: OverlappingRangeIPReservationList
+    plural: overlappingrangeipreservations
+    singular: overlappingrangeipreservation
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
+          API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: OverlappingRangeIPReservationSpec defines the desired state
+              of OverlappingRangeIPReservation
+            properties:
+              containerid:
+                type: string
+              ifname:
+                type: string
+              podref:
+                type: string
+            required:
+            - podref
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,196 @@
+---
+# Whereabouts: cluster-wide IPAM for Multus attachments.
+#
+# WHY THIS EXISTS
+#
+# The attachments in this cluster lease their addresses by DHCP
+# (cni-dhcp-daemon.yaml). That works where a DHCP server is on the wire —
+# v30iot has one. It does not work on the Ceph SAN (VLAN 49), which carries
+# Ceph-internal traffic and nothing else, by design: there is no DHCP server
+# there and there must not be one.
+#
+# `host-local`, already installed by cni-plugins-installer.yaml, cannot stand
+# in. It keeps its allocation state per node with no cluster-wide
+# coordination, so on a shared L2 segment two nodes hand out the same address.
+# Whereabouts coordinates through the Kubernetes API instead — allocations
+# live in the ippools/overlappingrangeipreservations CRDs, so nothing about
+# the address assignment touches the segment being addressed. That property is
+# what makes it usable on an isolated SAN.
+#
+# HOST ACCESS
+#
+# The DaemonSet runs with hostNetwork and privileged. That is how upstream
+# ships it, and it is what the three CNI components already here do —
+# multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml
+# all carry both. The reason is the same in each case: the pod writes a CNI
+# binary and its config into /opt/cni/bin and /etc/cni/net.d on the node,
+# which is node plumbing rather than a workload, and it must work before the
+# attachment it provides addresses for exists.
+#
+# The reconciler is deliberately NOT in here. It needs neither, and runs as an
+# ordinary pod-network Deployment — see whereabouts-reconciler.yaml.
+#
+# Vendored from upstream v0.9.4 (doc/crds/daemonset-install.yaml), with the
+# image tag pinned; upstream ships `:latest`. The CRDs it needs are the three
+# whereabouts-crd-*.yaml files alongside this one, vendored unmodified so an
+# upstream diff stays readable.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: whereabouts
+  namespace: kube-system
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: whereabouts
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: whereabouts-cni
+subjects:
+  - kind: ServiceAccount
+    name: whereabouts
+    namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: whereabouts-cni
+rules:
+  - apiGroups:
+      - whereabouts.cni.cncf.io
+    resources:
+      - ippools
+      - overlappingrangeipreservations
+      - nodeslicepools
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+  - apiGroups:
+      - coordination.k8s.io
+    resources:
+      - leases
+    verbs:
+      - '*'
+  - apiGroups: [""]
+    resources:
+      - pods
+    verbs:
+      - list
+      - watch
+      - get
+  - apiGroups: [""]
+    resources:
+      - nodes
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - network-attachment-definitions
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+      - get
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: whereabouts-config
+  namespace: kube-system
+  annotations:
+    kubernetes.io/description: |
+      Configmap containing user customizable cronjob schedule
+data:
+  # Read by the reconciler (whereabouts-reconciler.yaml), not by this
+  # DaemonSet. Governs how often stale allocations are swept — the case that
+  # matters here is a node lost hard enough that the CNI DEL never ran, which
+  # otherwise leaves an address held by a pod that no longer exists.
+  cron-expression: "30 4 * * *"
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: whereabouts
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts
+spec:
+  selector:
+    matchLabels:
+      name: whereabouts
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts
+        name: whereabouts
+    spec:
+      hostNetwork: true
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node, so the NoSchedule toleration
+      # is not optional — without it this schedules nowhere.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts
+          command: ["/bin/sh"]
+          args:
+            - -c
+            - |
+              SLEEP=false source /install-cni.sh
+              /token-watcher.sh &
+              /ip-control-loop -log-level debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: NODENAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: spec.nodeName
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "100Mi"
+            limits:
+              cpu: "100m"
+              memory: "200Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cnibin
+              mountPath: /host/opt/cni/bin
+            - name: cni-net-dir
+              mountPath: /host/etc/cni/net.d
+      volumes:
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: cni-net-dir
+          hostPath:
+            path: /etc/cni/net.d
@@ -0,0 +1,75 @@
+---
+# Sweeps whereabouts allocations that no live pod holds any more.
+#
+# The normal release path is the CNI DEL when a pod goes away. It does not run
+# when a node is lost hard — power cut, kernel panic, a reboot that never
+# comes back — and the allocation then stays claimed by a pod that no longer
+# exists. On a small range that matters: the Ceph SAN is a /29, so a handful
+# of leaked addresses is the difference between an OSD getting an address and
+# not.
+#
+# Runs as an ordinary pod-network Deployment. Unlike the DaemonSet next to it,
+# this needs no host access at all — it only talks to the Kubernetes API,
+# where the allocations live.
+#
+# replicas stays at 1: there is no leader election, so a second replica would
+# reconcile the same records concurrently.
+#
+# Vendored from upstream v0.9.4 (doc/crds/reconciler-deployment.yaml) with the
+# image tag pinned; upstream ships `:latest`. The schedule comes from the
+# whereabouts-config ConfigMap in whereabouts-daemonset.yaml.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: whereabouts-reconciler
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts-reconciler
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: whereabouts-reconciler
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts-reconciler
+        name: whereabouts-reconciler
+    spec:
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts-reconciler
+          command: ["/ip-reconciler"]
+          args:
+            - -log-level
+            - debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "50Mi"
+            limits:
+              cpu: "100m"
+              memory: "50Mi"
+          volumeMounts:
+            - name: cron-scheduler-configmap
+              mountPath: /cron-schedule
+      volumes:
+        - name: cron-scheduler-configmap
+          configMap:
+            name: whereabouts-config
+            defaultMode: 0744
+            items:
+              - key: "cron-expression"
+                path: "config"
@@ -42,12 +42,28 @@ spec:
      # the agent-injector mutating webhook is unused surface.
      enabled: false
    server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
      ha:
        enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
        raft:
          enabled: true
          setNodeId: true
@@ -65,8 +81,32 @@ spec:
              cluster_address = "[::]:8201"
            }

+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
            storage "raft" {
              path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
            }

            service_registration "kubernetes" {}
@@ -53,7 +53,13 @@ spec:
              command: ["/bin/sh", "-ec"]
              args:
                - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                  BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                    role=openbao-snapshot \
                    jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")
@@ -203,16 +203,16 @@ metadata:
  labels:
    app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
+  # Daily at 06:45 local (next free stagger slot). The app PVCs live in
+  # the home-assistant namespace regardless of which app/ dir owns
  # their manifest (matter-server, otbr, whisper, speech-to-phrase,
  # zone-configurator all set `namespace: home-assistant` via their
  # Kustomization, not per-resource): home-assistant-config (.storage
  # entity registry, recorder DB, secrets.yaml), esphome-data,
  # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
+  # speech-to-phrase-models, speech-to-phrase-train, zigbee2mqtt-data,
+  # zwavejs-store, zone-configurator-config. On restore,
+  # home-assistant-config has its
  # own path (the HA pod's `seed` initContainer pulling the native HAOS
  # backup from Garage, see the home-assistant Kustomization comment in
  # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
@@ -229,6 +229,20 @@ spec:
    storageLocation: default
    ttl: 720h0m0s
    snapshotMoveData: true
+    # The ha-pg CNPG cluster now runs in this namespace (it did not when
+    # this schedule was written — the old comment here still claimed
+    # "no CNPG cluster in this namespace"). Its PVC ha-pg-1 carries the
+    # cnpg.io/cluster label and was being swept in: a crash-consistent
+    # snapshot of a running Postgres data directory is not a restorable
+    # backup, the same trap the wordpress-daily comment below describes.
+    # ha-pg is covered by Barman — daily base backup at 02:50 UTC
+    # (kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml) plus
+    # continuous WAL archiving — so this exclusion removes a redundant
+    # 20 GiB upload, matching paperless/authentik/immich/trmnl above.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule
@@ -8,6 +8,17 @@ spec:
    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
+  # Do not start restarting OSDs while PGs are still recovering. The default
+  # is false, so Rook would proceed regardless — and paired with the old
+  # osdMaxUpdatesInParallel default that means it can take the next OSD down
+  # while the previous one's data is still being caught up.
+  #
+  # With three OSDs and size 3 there is no spare copy to fall back on: the
+  # gate between one OSD returning and the next going down is the only thing
+  # that keeps a rolling change from becoming an outage. See
+  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
+  # from the other side.
+  upgradeOSDRequiresHealthyPGs: true
  mon:
    count: 3
    allowMultiplePerNode: false
@@ -36,26 +47,27 @@ spec:
  # (paperless + audiobookshelf + immich + ... all clone at once).
  cephConfig:
    mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than left out. Removing the line would
+      # not restore it: `ceph config` keeps a value that only disappears from
+      # this file, so the old 20 would sit in the mon config database
+      # indefinitely — the same trap the bluestore settings below describe,
+      # and the one that crash-looped an OSD during the network provider
+      # change when public_network survived being deleted here.
      #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
+      # It was 20 because the mons share the node's /var with containerd, and
+      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
+      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
+      # the image cache filled and GC could not reclaim it (#434).
+      #
+      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
+      # used, so 30% free is a real reserve again and the ~106 MB mon database
+      # is irrelevant against it. Reaching 70% would be a genuine signal about
+      # the node rather than noise about the image cache.
+      #
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules still watches /var directly for the real risk, which
+      # is the node filling rather than the mon growing.
+      mon_data_avail_warn: "30"
      # Stop compacting the whole RocksDB prefix on every paxos trim.
      #
      # Upstream default is true — "Compact a certain prefix (including paxos)
@@ -147,19 +159,65 @@ spec:
    ssl: false
  monitoring:
    enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  #
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  #
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
+  #
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
+  #
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
+  #
+  # ISOLATION IS PRESERVED
+  #
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
+  #
+  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
+  # reproducible from the manifests alone. Removing addressRanges here does
+  # not clear public_network and cluster_network from Ceph's own config
+  # database — the same trap the bluestore comment above describes: `ceph
+  # config` retains a value that is only deleted from this file. During the
+  # phase A rollout the first OSD came up on the pod network, still had
+  # public_network 10.2.40.0/24 in the config database, found no interface in
+  # it and crash-looped on "Failed to pick public address". Both keys were
+  # removed with `ceph config rm global public_network` and the same for
+  # cluster_network; Rook then wrote cluster_network afresh from the
+  # attachment below. Anyone rebuilding this cluster from scratch will not hit
+  # it — the config database starts empty — but anyone repeating a provider
+  # change on a live cluster will.
+  #
+  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
+  # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
  network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
  # taken out of the storage fleet by removing it here. Under useAllNodes the
  # operator re-provisions an OSD on any node whose device still carries a
@@ -193,6 +251,15 @@ spec:
  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
  #   S437NF0M501176    PM983     — this OSD
  storage:
+    # One OSD restarted at a time. The default is 20, which on a three-OSD
+    # cluster means all of them at once — every PG inactive, every pool
+    # unreachable, for as long as the restart takes. The default suits a
+    # cluster large enough that 20 OSDs is a small fraction of it; here it is
+    # the whole thing.
+    #
+    # This binds any change that rewrites the OSD deployment spec, not just
+    # Ceph version upgrades — a network provider change is one such.
+    osdMaxUpdatesInParallel: 1
    useAllNodes: false
    useAllDevices: false
    nodes:
@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
  - cephcluster.yaml
  - toolbox.yaml
  - storageclass.yaml
@@ -0,0 +1,84 @@
+---
+# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
+# without every Ceph daemon being put on host networking to get there.
+#
+# WHAT THIS REPLACES, AND WHY
+#
+# `addressRanges` only takes effect under `provider: host` — Rook ignores it
+# for the default CNI provider. So carrying replication over the dedicated
+# 10G link meant setting provider: host, which is not per-daemon: it puts
+# mons on host networking too. A mon's address is then the node's DHCP lease,
+# and when that lease moves the mon cannot bind and fails. Worse, three mons
+# on three nodes each hold the mon host ports, so Rook cannot schedule a
+# replacement anywhere and the failover it starts cannot finish.
+#
+# None of that has anything to do with the goal. Per the Ceph documentation
+# the cluster network carries OSD heartbeat, replication and recovery only —
+# "Ceph Monitors always operate on the public network". Attaching it here
+# gives the OSDs the link and leaves everything else on the pod network,
+# where Rook manages identity itself.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.1",
+        "range_end": "10.99.0.6"
+      }
+    }
@@ -20,16 +20,39 @@ machine:
  #                            the MS-01 mesh later needs neither a new image
  #                            nor a reinstall — which is exactly the property
  #                            lost by installing without it.
+  #   siderolabs/i915          Intel GPU microcode and kernel modules for the
+  #                            Alder Lake-P Iris Xe in the MS-01. No workload
+  #                            requests /dev/dri today; it is here for the same
+  #                            reason as thunderbolt, so that wanting hardware
+  #                            transcoding later is a config change rather than
+  #                            another rolling upgrade of every node.
  #
-  # a21cec8f… is the bare-metal schematic described in RUNBOOK-cold-rebuild.md:
-  # same net.ifnames=1 as the Proxmox one (the predictable interface names the
-  # per-node VLAN parents match against), minus qemu-guest-agent.
+  # Not included, and why, so the absences are not mistaken for oversights:
+  # the X710 (i40e) and I226 (igc) NICs need no firmware extension —
+  # intel-ice-firmware covers E810/ice only. mei is a prerequisite for Intel
+  # Arc discrete GPUs, of which there are none. iscsi-tools, nvme-cli and
+  # util-linux-tools have no consumer: storage is Ceph RBD, and device health
+  # is reported by Ceph itself rather than a node-level exporter.
+  #
+  # 0f6fb8bd… is the bare-metal schematic: same net.ifnames=1 as the Proxmox
+  # one (the predictable interface names the per-node VLAN parents match
+  # against), minus qemu-guest-agent. It supersedes a21cec8f…, which was
+  # identical but without i915 — see RUNBOOK-cold-rebuild.md.
  #
  # Only read at install and upgrade time. Changing it here does nothing to a
  # running node until that node is upgraded. Bump the tag with the Talos
  # version — it must match the running version, or the upgrade is a downgrade.
+  #
+  # Upgrades follow the documented path: migration is only tested between
+  # adjacent minor releases, so go to the latest patch of each intermediate
+  # minor rather than skipping. 1.12.11 -> 1.13.9 satisfies that — one minor
+  # step from the last 1.12 patch, no intermediate needed.
+  #
+  # Kubernetes is a separate decision and is not forced by this. Talos 1.13
+  # supports Kubernetes 1.36, 1.35, 1.34, 1.33, 1.32 and 1.31, so the cluster
+  # stays on 1.35.2 until `talosctl upgrade-k8s` moves it deliberately.
  install:
-    image: factory.talos.dev/metal-installer/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64:v1.12.6
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
  features:
    rbac: true
    stableHostname: true
@@ -20,6 +20,29 @@ machine:
    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
    # This disables v6 entirely on that link, including link-local.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument, applied to the physical ports rather than a VLAN child.
+    #
+    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
+    # enp87s0 is unused. Both sit on the untagged fallback network, where by
+    # design nothing is supposed to run — and both were autoconfiguring
+    # themselves from its router advertisements: an fd00::/64 ULA each, plus
+    # an IPv6 default route each.
+    #
+    # The result was four IPv6 default routes per node at identical priority
+    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
+    # globally routable source address. With nothing to break the tie,
+    # outbound IPv6 mostly left through an interface whose only usable source
+    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
+    # normally. It surfaced as two NetBird probes reporting the proxies down
+    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
+    # outside.
+    #
+    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
+    # parent leaves the VLAN child alone. mgmt0 kept its global address and
+    # its fd40:: address, Ceph did not notice, and every failing probe
+    # recovered on the next scrape.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
@@ -94,6 +117,30 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# Management addressing is DHCP and the lease is NOT reserved anywhere in this
+# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
+# and opentofu/unifi.tf declares no live resources. The addresses do move under
+# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
+# without its Node object ever re-registering (issue #635).
+#
+# kube-apiserver does not follow when that happens. Talos starts it with
+# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
+# so the advertised address is fixed at pod start and the default lease
+# reconciler goes on renewing the masterlease under the OLD address. Nothing
+# expires it while the process lives, so `default/kubernetes` keeps handing out
+# a dead endpoint and the node's live address is never registered at all — 19h
+# of TargetDown on job="apiserver", a monitoring blind spot on one of three
+# apiservers, and it cleared only when the static pod restarted.
+#
+# So after any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide`, and restart the offending node
+# (`talosctl -n <node> reboot`) if it does not.
+#
+# Pinning mgmt0 with a static AddressConfig alongside this document would remove
+# the trigger, but it needs an address outside the UDM's DHCP pool — which is
+# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
+# prove it with `talosctl apply-config --mode=try` before making it permanent.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -117,13 +164,20 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.1/29, because reaching this link required
+# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
+# host networking, mons included. Nothing needs it now, and by the rule this
+# segment is built on — Ceph-internal traffic and nothing else — the node has
+# no business on it either.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29
@@ -9,6 +9,15 @@ machine:
    # macvlan parent for the v30iot pod attachments; host addresses on that L2
    # make node replies bypass the UDM and break every UDM-DNATed flow.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are visible on bare metal. Select the installer target
    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -89,10 +98,17 @@ up: true
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
 # untagged on the wire. OSD<->OSD replication only. Not routed off its own
 # segment and deliberately has no default route.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.2/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29
@@ -8,6 +8,15 @@ machine:
  sysctls:
    # v30iot NIC carries no host addressing — see k8s-1.yaml.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are visible on bare metal. Select the installer target
    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -73,10 +82,16 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml.
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.3/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29