fix(multus): drop runtime apk add from the CNI plugins installer

Owner directive: tools belong in the image, not installed at container
start. This is the highest-blast-radius instance of the pattern left in the
repo — a DaemonSet, so it runs on every node.

No image was needed. `apk add --no-cache curl tar` turns out to have been
unnecessary from the start: verified in alpine:3.21 that `tar` is already
busybox (v1.37.0) and handles -xzf, and that busybox `wget` follows
GitHub's release redirect to objects.githubusercontent.com over TLS using
the base image's /etc/ssl/certs/ca-certificates.crt. Only `curl` is
genuinely absent, and wget covers what it was used for. So this replaces
the install with `wget -q -O` rather than introducing another layered
image.

Severity note for the record: the apk add sat inside the else branch, so
steady-state nodes never reached it. It ran when plugins were absent — i.e.
on a fresh node or after /opt/cni/bin was wiped, which is precisely the
moment you least want a dependency on package-index availability.

Verified by running the RENDERED init script in alpine:3.21 against the
real release URL, both paths:

  empty dir     -> Fetching ...; Installed: host-local, macvlan, static
                   macvlan executable, /opt/cni/bin owner 0:0, exit 0
  already there -> "CNI plugins already installed (CNI macvlan plugin
                   v1.5.1)", exit 0, no fetch

The second line is worth noting: that version string comes from executing
the extracted binary, so the download and extraction are confirmed good,
not merely present. The chown 0:0 also still applies, which the existing
comment explains is required or Cilium's apply-sysctl-overwrites init
container fails with "Permission denied".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/alert-monitor.yml

@@ -30,19 +30,19 @@
 # Triggering:
 #   - cron   : a cron NAMED exactly "alert-monitor" (name filter, same
 #              mechanism as update-audit.yml):
-#                woodpecker repo cron add --repo konairius/homelab \
+#                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name alert-monitor --branch master --schedule '*/15 * * * *'
-#   - deploy : woodpecker-cli pipeline deploy konairius/homelab <n> alert-monitor
-#              where <n> is an existing pipeline to promote.
+#   - manual : woodpecker pipeline create --branch master \
+#                --var PIPELINE=alert-monitor k.renner/homelab
 when:
   - event: cron
     cron: alert-monitor
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "alert-monitor"'
+  - event: manual
+    evaluate: 'PIPELINE == "alert-monitor"'
 
 steps:
   - name: fetch-alerts
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key
@@ -101,7 +101,7 @@ steps:
         bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token
@@ -130,43 +130,13 @@ steps:
              | select([.receivers[]?.name] | (length == 0) or any(. != "null"))]' \
           .ci-alert-monitor/alerts-raw.json > .ci-alert-monitor/alerts-active.json
         echo "Active alerts (after heartbeat filter): $$(jq length .ci-alert-monitor/alerts-active.json)"
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
         curl -sfL "$$API/issues?state=open&type=issues&limit=100" > .ci-alert-monitor/open-issues.json
         TITLES=$$(jq -c '[.[].title]' .ci-alert-monitor/open-issues.json)
         NOISE_BODY=$$(jq -r 'map(select(.title == "Alert monitor: noise log")) | .[0].body // ""' \
           .ci-alert-monitor/open-issues.json)
-        # Suppression entries are exactly the structured lines the write-back
-        # step emits: "- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>". Parse THAT
-        # shape rather than grepping the whole body for 16-hex tokens. A reason
-        # routinely cites another alert's fingerprint in prose ("see the issue
-        # filed for fingerprint <fp>"), and the body-wide grep turned every such
-        # citation into a silent, permanent suppression of an alert that was
-        # never classified as noise — verified against the live noise log, which
-        # yielded 22 fingerprints for 21 real entries, the extra one being
-        # 2ddc41edb689fc33, an alert that HAD been filed as a genuine issue.
-        #
-        # Entries also expire after NOISE_TTL_DAYS. A noise verdict says "this
-        # occurrence was benign", not "this alert can never be interesting
-        # again" — but the fingerprint is sha256(alertname|scope) and therefore
-        # stable across recurrences, so without an expiry a single blip mutes
-        # that alertname+scope pair forever. Re-triaging a recurrence weeks
-        # later is cheap; missing a real outage because a July blip muted
-        # openbao's KubePodNotReady is not.
-        #
-        # Lines that fail to parse are simply dropped, i.e. they stop
-        # suppressing — this fails toward MORE alerting, never less.
-        #
-        # date is BusyBox here, not GNU: `date -d '30 days ago'` errors out, so
-        # the cutoff goes through epoch arithmetic and `date -d @<epoch>`, which
-        # this image does support.
-        NOISE_TTL_DAYS=$${NOISE_TTL_DAYS:-30}
-        CUTOFF=$$(date -u -d @$$(( $$(date -u +%s) - NOISE_TTL_DAYS * 86400 )) +%F)
-        FPS=$$(printf '%s\n' "$$NOISE_BODY" \
-          | sed -nE 's/^- ([0-9a-f]{16}) [^ ]+ — ([0-9]{4}-[0-9]{2}-[0-9]{2}):.*/\2 \1/p' \
-          | awk -v c="$$CUTOFF" 'BEGIN{gsub(/-/,"",c)} { d=$$1; gsub(/-/,"",d); if (d+0 >= c+0) print $$2 }' \
+        FPS=$$(printf '%s' "$$NOISE_BODY" | grep -oE '\b[0-9a-f]{16}\b' \
           | jq -R -s -c 'split("\n") | map(select(length > 0))')
-        echo "Noise suppressions in effect: $$(printf '%s' "$${FPS:-[]}" | jq length)" \
-             "(TTL $${NOISE_TTL_DAYS}d, entries dated >= $$CUTOFF)"
         # Synthesize pseudo-alerts for BLIPS: lookback series whose label set
         # is not currently firing at all (checked against the PRE-filter raw
         # snapshot, so live-but-null-routed alerts don't reappear as blips).
@@ -238,7 +208,7 @@ steps:
           exit 0
         fi
         jq . /tmp/alert-verdicts.json > /dev/null   # fail loudly on malformed output
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
         STAMP=$$(date -u +%Y-%m-%d)
 
         gitea_write() {
@@ -290,7 +260,7 @@ steps:
               '{body: ($$old + "\n" + $$add)}' > /tmp/gitea-patch.json
             gitea_write PATCH "$$API/issues/$$NOISE_NUM" /tmp/gitea-patch.json
           else
-            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
+            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
             jq -n --rawfile hdr /tmp/noise-header.txt --rawfile add /tmp/noise-lines.txt \
               '{title: "Alert monitor: noise log", body: ($$hdr + $$add)}' > /tmp/gitea-create.json
             gitea_write POST "$$API/issues" /tmp/gitea-create.json

.woodpecker/build-ci-images.yml

@@ -1,17 +1,6 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
-# The kaniko image is NOT Google's. Google archived
-# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
-# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
-# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
-# of its own, so these come from gitlab-ci-utils, which builds that fork for
-# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
-# and is deliberately unused; everything else in this repo is pinned too.
-#
-# The executor path, flags and busybox layout are unchanged from Google's
-# image, which is why the swap touched nothing but the `image:` lines (#552).
-#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -19,42 +8,32 @@
 # the admin account could (#330).
 #
 # The token is still OWNED by gitea_admin, because Gitea scopes packages to
-# a user or org namespace and most of these images live under `gitea_admin/*`.
-# The rebuild-critical ones additionally publish to `konairius/` — an org
-# created for the ghcr mirror path, with gitea_admin as a member so this same
-# token can push there. The full gitea_admin/* -> org migration is a
-# 38-reference change tracked separately in #330 rather than bundled here.
-# Note this pipeline runs no LLM agent, so the exfil path that motivated the
-# other cutovers does not apply to it.
-#
-# Push builds run on master only. Without the branch filter a feature branch
-# that touches this file started all steps at once: it pushed unreviewed
-# images and used the whole woodpecker-ci-quota, so the PR checks could not
-# get a pod (REN-286).
-#
-# Steps run in sequence (no depends_on): a change to this file rebuilds every
-# image, and in parallel they exceed the woodpecker-ci-quota.
+# a user or org namespace and these images live under `gitea_admin/*` — a
+# different account cannot publish there. Moving them to a dedicated org is
+# the remaining cleanup; it is a 38-reference migration across pipelines and
+# Flux-deployed CronJobs, so it is tracked separately in #330 rather than
+# bundled here. Note this pipeline runs no LLM agent, so the exfil path that
+# motivated the other cutovers does not apply to it.
 when:
   - event: push
-    branch: master
     path:
       - "ci/images/**"
       - "secret-rotator/**"
-      - "review-bot/**"
       - ".woodpecker/build-ci-images.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-ci-images"'
+  - event: manual
+    evaluate: 'PIPELINE == "build-ci-images"'
 
 steps:
   - name: build-ansible
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: ci/images/ansible
+    depends_on: []
     # Only rebuild when THIS image's source (or the pipeline file) changed, or
     # on a manual run. Without this every step ran on any matching push, so a
     # secret-rotator commit needlessly rebuilt all four ~1.3G ci-* images and
     # orphaned ~5G of registry layers per run — a main driver of #336.
     when:
-      - event: deployment
+      - event: manual
       - event: push
         path: ["ci/images/ansible/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -74,10 +53,11 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-ansible:latest
 
   - name: build-k8s
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: ci/images/k8s
+    depends_on: []
     when:
-      - event: deployment
+      - event: manual
       - event: push
         path: ["ci/images/k8s/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -96,34 +76,12 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:$TAG \
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
-  - name: build-vmtest
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/vmtest
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/vmtest/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-vmtest:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-vmtest:latest
-
   - name: build-tofu
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: ci/images/tofu
+    depends_on: []
     when:
-      - event: deployment
+      - event: manual
       - event: push
         path: ["ci/images/tofu/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -150,10 +108,11 @@ steps:
   # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
   # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
   - name: build-secret-rotator
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: secret-rotator
+    depends_on: []
     when:
-      - event: deployment
+      - event: manual
       - event: push
         path: ["secret-rotator/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -172,333 +131,31 @@ steps:
           --destination gitea.renner.uno/gitea_admin/secret-rotator:$TAG \
           --destination gitea.renner.uno/gitea_admin/secret-rotator:latest
 
-  # review-bot: the Go service that reviews pull requests with a Mistral agent (webhook plus sweep).
-  # Runs after Gitea is up, so no ghcr mirror.
-  - name: build-review-bot
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: review-bot
-    when:
-      - event: deployment
-      - event: push
-        path: ["review-bot/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d-%H%M)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/review-bot:$TAG \
-          --destination gitea.renner.uno/gitea_admin/review-bot:latest
-
   # Runtime image, not CI tooling: a general-purpose alpine + jq base for
   # workloads that need to read JSON from a shell script. Named for what it IS,
   # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
-  #
-  # Published THREE times, and the third one is not a typo. Its consumers — the
-  # zigbee2mqtt seed-data init container and the trmnl renderer — start while
-  # the cluster is still being rebuilt, and at that moment gitea.renner.uno does
-  # not exist, because Gitea runs inside the cluster being built. A cold-rebuild
-  # rehearsal caught both in ImagePullBackOff.
-  #
-  # So there is a copy on ghcr.io. But the manifests must NOT point at it:
-  # normal operation pulls from the local registry, and an external reference
-  # would make every steady-state pull leave the network. The recovery source
-  # and the everyday source are different things, exactly as they are for Flux's
-  # git source.
-  #
-  # Talos resolves that with machine.registries.mirrors, which takes an ordered
-  # endpoint list — gitea first, ghcr second (bootstrap.sh). Containerd keeps
-  # the repository path when it falls through, so the two registries have to
-  # agree on it: `gitea_admin/alpine-jq` cannot be served by ghcr, where the
-  # namespace is fixed to the account name. Hence the second Gitea destination
-  # under konairius/, which the manifests reference and which matches the ghcr
-  # path exactly.
-  #
-  # The gitea_admin/ copy stays until nothing references it any more.
   - name: build-alpine-jq
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: ci/images/alpine-jq
+    depends_on: []
     when:
-      - event: deployment
+      - event: manual
       - event: push
         path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
     environment:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest \
-          --destination gitea.renner.uno/konairius/alpine-jq:$TAG \
-          --destination gitea.renner.uno/konairius/alpine-jq:latest \
-          --destination ghcr.io/konairius/alpine-jq:$TAG \
-          --destination ghcr.io/konairius/alpine-jq:latest
-
-  # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
-  # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
-  # node-side fetch from GitHub that had NO checksum verification and sat on the
-  # critical path of every node rebuild. The download now happens here, once,
-  # where the sha256 is checked — see the Containerfile for the full rationale.
-  #
-  # This one ALSO publishes to ghcr.io. A cold-rebuild rehearsal proved why: the
-  # DaemonSet could not pull from gitea.renner.uno, because Gitea runs inside the
-  # very cluster being rebuilt. Without the plugins there is no macvlan, so
-  # Multus cannot attach ceph-san, so Rook's network canary never starts, so the
-  # CephCluster sits at "Configuring Ceph Mons" — and OpenBao waits forever for a
-  # PVC that will never bind. Moving the download here fixed the integrity gap
-  # but quietly swapped github.com on the critical path for something strictly
-  # worse: a registry that by definition does not exist during a recovery. The
-  # GHCR copy is public, so a bare node can pull it with no credential and no
-  # Gitea.
-  #
-  # The test that proved this also proved the rule is not specific to this
-  # image. alpine-jq and alpine-openssl mirror for the same reason — see the
-  # note on build-alpine-jq. Anything an app needs while the cluster is still
-  # being built cannot live only in that cluster's own registry.
-  - name: build-cni-plugins
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/cni-plugins
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest \
-          --destination gitea.renner.uno/konairius/cni-plugins:$TAG \
-          --destination gitea.renner.uno/konairius/cni-plugins:latest \
-          --destination ghcr.io/konairius/cni-plugins:$TAG \
-          --destination ghcr.io/konairius/cni-plugins:latest
-
-  # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
-  # needing s_client/x509. Named for what it is; first consumer is the paperless
-  # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
-  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
-  # base, so this one needs an image rather than a busybox substitute.
-  #
-  # Mirrored to ghcr.io for the reason given on build-alpine-jq: it is an init
-  # container, so paperless does not start without it, and during a rebuild
-  # gitea.renner.uno is not there to serve it. A cold-rebuild rehearsal left
-  # paperless in Init:ErrImagePull on exactly this image.
-  - name: build-alpine-openssl
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/alpine-openssl
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest \
-          --destination gitea.renner.uno/konairius/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/konairius/alpine-openssl:latest \
-          --destination ghcr.io/konairius/alpine-openssl:$TAG \
-          --destination ghcr.io/konairius/alpine-openssl:latest
-
-  # Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
-  # run as two services on edge-1. Replaces pip-into-venv on the host, where an
-  # unpinned VCS requirement re-cloned and restarted the heating bridge on every
-  # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
-  # builds the same way and lands in the same registry so the audit sees it.
-  - name: build-heating-can
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/heating-can
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/heating-can/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
-          --destination gitea.renner.uno/gitea_admin/heating-can:latest
-
-  # netbird-service-sync: the controller that reconciles NetBird reverse-proxy
-  # services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
-  # Runtime artifact like heating-can; same registry so the audit tracks it.
-  - name: build-netbird-service-sync
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/netbird-service-sync
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/netbird-service-sync/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/netbird-service-sync:$TAG \
-          --destination gitea.renner.uno/gitea_admin/netbird-service-sync:latest
-
-  # authentik-gate-sync: the authentik-gate ingress class (Ingress -> Authentik
-  # proxy gate + Traefik route). Its unit tests run as a stage of the image build.
-  - name: build-authentik-gate-sync
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/authentik-gate-sync
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/authentik-gate-sync/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      # YYYY-MM-DD-HHMM: the image-automation ImagePolicy picks the newest.
-      - TAG=$(date +%Y-%m-%d-%H%M)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/authentik-gate-sync:$TAG \
-          --destination gitea.renner.uno/gitea_admin/authentik-gate-sync:latest
-
-  # kuma-sync: reconciles KumaMonitor resources into Uptime Kuma. Its unit
-  # tests run as a stage of the image build.
-  - name: build-kuma-sync
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/kuma-sync
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/kuma-sync/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      # YYYY-MM-DD-HHMM: the image-automation ImagePolicy picks the newest.
-      - TAG=$(date +%Y-%m-%d-%H%M)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/kuma-sync:$TAG \
-          --destination gitea.renner.uno/gitea_admin/kuma-sync:latest
-
-  # bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
-  # artifact like heating-can; same registry so the audit tracks it.
-  - name: build-bird
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/bird
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/bird:$TAG \
-          --destination gitea.renner.uno/gitea_admin/bird:latest
-
-  # Orchestrator for the monthly backup-verify CronJob (velero ns).
-  - name: build-backup-verify
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/backup-verify
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/backup-verify/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-backup-verify:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-backup-verify:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest

.woodpecker/ci-claude-refresh.yml

@@ -14,11 +14,11 @@
 #                 and  ci-claude:latest.  The Containerfile makes this layer
 #                 byte-reproducible, so rebuilding an unchanged version dedups
 #                 in the registry instead of orphaning a fresh ~260 MB copy.
-#   3. bump-pr  — (only when any pin is off the latest version) rewrite every
-#                 consumer pin to  ci-claude:<version>  and open a PR. A human
+#   3. bump-pr  — (only when the version actually changed) rewrite the 5
+#                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -48,14 +48,14 @@
 # OWNER SETUP — the token + secret above are already provisioned. The one
 # remaining step, to be run AFTER this merges to master (a cron references a
 # pipeline that must exist on the target branch), is to register the schedule:
-#   woodpecker repo cron add --repo konairius/homelab \
+#   woodpecker repo cron add --repo k.renner/homelab \
 #     --name ci-claude-refresh --branch master --schedule '@weekly'
 # Weekly is plenty; the pipelines tolerate being a few days behind a release.
 #
 # Triggering:
 #   - cron   : the cron NAMED exactly "ci-claude-refresh" (see above).
-#   - deploy : woodpecker-cli pipeline deploy konairius/homelab <n> ci-claude-refresh
-#              where <n> is an existing pipeline to promote.
+#   - manual : woodpecker pipeline create --branch master \
+#                --var PIPELINE=ci-claude-refresh k.renner/homelab
 #   - push   : a change to the Containerfile (or this file) ON master forces a
 #              rebuild of the current latest version so Containerfile fixes ship
 #              promptly. Restricted to master ON PURPOSE — a feature branch must
@@ -65,8 +65,8 @@
 when:
   - event: cron
     cron: ci-claude-refresh
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "ci-claude-refresh"'
+  - event: manual
+    evaluate: 'PIPELINE == "ci-claude-refresh"'
   - event: push
     branch: master
     path:
@@ -84,45 +84,44 @@ steps:
       - |
         set -eu
         mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image, DISCOVERED rather than listed. This used to
-        # be a hardcoded list carrying the instruction "keep this in sync with
-        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
-        # missing, so they sat on a version no bump could ever reach. A list
-        # that must be maintained by hand is the bug; deriving it removes the
-        # failure mode instead of correcting one instance of it.
-        #
-        # Matches a version-pinned reference only, which is what excludes this
-        # file: its own steps run ci-claude:latest, and :latest never needs
-        # rewriting. That exclusion is load-bearing — do not write a literal
-        # ci-claude version anywhere in THIS file, not even in prose, or it
-        # adds itself to the consumer set and the sed below rewrites it. The
-        # tag-scheme note at the top says <claude-version> for that reason.
-        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
-        # An empty list would make the grep below read stdin and hang forever
-        # rather than fail, so it is a hard stop.
-        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
-        echo "consumers: $$CONSUMERS"
+        # Files that pin the image. Keep this list in sync with the consumers.
+        CONSUMERS="
+          .woodpecker/security-scan.yml
+          .woodpecker/alert-monitor.yml
+          .woodpecker/update-audit.yml
+          .woodpecker/pr-review.yml
+          .woodpecker/issue-fix.yml
+        "
         NEW=$$(npm view @anthropic-ai/claude-code version)
         case "$$NEW" in
           [0-9]*.[0-9]*.[0-9]*) : ;;
           *) echo "npm returned an implausible version '$$NEW' — aborting."; exit 1 ;;
         esac
-        # Distinct tags the consumers currently reference. Divergent pins are
-        # normal (a pipeline bumped on its own); the bump rewrites all of them.
-        CUR=$$(grep -hoE 'ci-claude:[0-9][0-9A-Za-z._-]*' $$CONSUMERS | sed 's/^ci-claude://' | sort -u | tr '\n' ' ' | sed 's/ $$//')
-        [ -n "$$CUR" ] || { echo "No ci-claude pin found in the consumers — aborting."; exit 1; }
-        echo "latest=$$NEW  current=[$$CUR]  event=$$CI_PIPELINE_EVENT"
+        # The tag every consumer currently references. They are kept identical,
+        # so this must resolve to EXACTLY ONE non-empty tag; anything else
+        # (no match, or a partial/disagreeing set) aborts rather than risk a
+        # corrupting sed later. Counting with `grep -c .` (non-empty lines)
+        # instead of `wc -l` is deliberate: an empty result yields 0 here but
+        # 1 under wc -l, which would slip past the guard.
+        CUR=$$(grep -hoE 'ci-claude:[0-9A-Za-z._-]+' $$CONSUMERS | sed 's/^ci-claude://' | sort -u)
+        NTAGS=$$(printf '%s' "$$CUR" | grep -c . || true)
+        if [ -z "$$CUR" ] || [ "$$NTAGS" -ne 1 ]; then
+          echo "Expected exactly one ci-claude tag across the consumers, got [$$CUR]."
+          echo "Refusing to guess — fix the pins by hand."; exit 1
+        fi
+        echo "latest=$$NEW  current=$$CUR  event=$$CI_PIPELINE_EVENT"
         printf '%s' "$$NEW" > "$$CI_WORKSPACE/.ci-refresh/version"
         printf '%s' "$$CUR" > "$$CI_WORKSPACE/.ci-refresh/current"
-        # Build when any pin is off latest, or on a push (Containerfile edit must
-        # ship even if the version is unchanged). Bump when any pin is off latest.
+        # Build when the version changed, or on a push (Containerfile edit must
+        # ship even if the version is unchanged). Bump the pins only when the
+        # version the consumers reference is not the latest.
         if [ "$$CUR" != "$$NEW" ] || [ "$$CI_PIPELINE_EVENT" = "push" ]; then
           touch "$$CI_WORKSPACE/.ci-refresh/do-build"; echo "will build $$NEW"
         else
           echo "already on $$NEW — nothing to build"
         fi
         if [ "$$CUR" != "$$NEW" ]; then
-          touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR [$$CUR] -> $$NEW"
+          touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR $$CUR -> $$NEW"
         else
           echo "consumers already pin $$NEW — no PR"
         fi
@@ -131,7 +130,7 @@ steps:
   #    --build-arg to the resolved version (not "latest") guarantees the tag and
   #    the image content agree even if a new release lands mid-run.
   - name: build
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: ci/images/claude
     depends_on: [decide]
     environment:
@@ -166,29 +165,27 @@ steps:
       - |
         [ -f "$$CI_WORKSPACE/.ci-refresh/do-bump" ] || { echo "no version change — no PR"; exit 0; }
         set -eu
-        # Secret values can carry a trailing newline; it breaks the push URL and the auth header.
-        PUSH_TOKEN=$$(printf '%s' "$$PUSH_TOKEN" | tr -d '\r\n')
-        [ -n "$$PUSH_TOKEN" ] || { echo "PUSH_TOKEN is empty — aborting"; exit 1; }
         NEW=$$(cat "$$CI_WORKSPACE/.ci-refresh/version")
         CUR=$$(cat "$$CI_WORKSPACE/.ci-refresh/current")
         # Both were validated non-empty by the decide step; re-assert before a
         # sed that would corrupt every pin if either were blank.
         [ -n "$$NEW" ] && [ -n "$$CUR" ] || { echo "empty version/current sentinel — aborting"; exit 1; }
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
         BRANCH="ci-image/claude-$$NEW"
-        if git ls-remote --heads "https://gitea.renner.uno/konairius/homelab.git" "$$BRANCH" | grep -q .; then
+        if git ls-remote --heads "https://gitea.renner.uno/k.renner/homelab.git" "$$BRANCH" | grep -q .; then
           echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
           exit 0
         fi
-        # Same discovery as the decide step — see the comment there for why this
-        # is not a maintained list.
-        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
-        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
-        # NEW goes into a sed replacement: allow only a plain x.y.z version.
-        printf '%s' "$$NEW" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$$' || { echo "refusing non-x.y.z version '$$NEW'"; exit 1; }
-        # Every version-pinned tag becomes NEW, whatever it was; consumers only.
+        CONSUMERS="
+          .woodpecker/security-scan.yml
+          .woodpecker/alert-monitor.yml
+          .woodpecker/update-audit.yml
+          .woodpecker/pr-review.yml
+          .woodpecker/issue-fix.yml
+        "
+        # Exact-string swap of the current tag for the new one, consumers only.
         for f in $$CONSUMERS; do
-          sed -i "s#ci-claude:[0-9][0-9A-Za-z._-]*#ci-claude:$$NEW#g" "$$f"
+          sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
         done
         # Author is claude-bot (the PAT's account); the commit subject makes
         # the automated-bump nature clear.
@@ -197,11 +194,11 @@ steps:
         git checkout -b "$$BRANCH"
         git add $$CONSUMERS
         git commit -m "chore(ci): bump ci-claude image to $$NEW"
-        git push "https://$$PUSH_TOKEN@gitea.renner.uno/konairius/homelab.git" HEAD:"$$BRANCH"
+        git push "https://$$PUSH_TOKEN@gitea.renner.uno/k.renner/homelab.git" HEAD:"$$BRANCH"
         {
           printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
           printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
           printf 'run the image so CI stops running a stale Claude Code.\n\n'
           printf 'Merging is a human decision — review, then merge to roll it out.\n'
         } > /tmp/pr-body.md

.woodpecker/ci-manifests-test.yml

@@ -1,343 +0,0 @@
-# Validate the GitOps tree and the pipeline definitions on PRs that touch them.
-#
-# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
-# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
-# so a broken manifest reached the cluster and was caught by Flux failing to
-# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would
-# have been caught here instead:
-#
-#   * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml,
-#     which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed
-#     it happily — it is valid YAML, just a map instead of a string — so
-#     "it parses" was mistaken for "it is valid". Only woodpecker lint catches it.
-#   * a rotator pinned to an image digest that no longer existed (#458).
-#   * manifests whose kustomize wiring was only ever checked by hand.
-#
-# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
-# here is a static check that either passes or fails on the checkout alone, so it
-# can gate PRs without becoming a bottleneck or a flake source.
-when:
-  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is two
-  # steps on already-cached images, so running it everywhere is free; this one
-  # pulls four images and builds every kustomization in the tree, which is real
-  # cost to put on a docs-only PR. The filter covers everything the steps actually read:
-  # kubernetes/** for the manifests, scripts and promtool suites, .woodpecker/**
-  # for the lint (and so an edit to this file re-runs its own checks), and the
-  # promtool driver script by name.
-  #
-  # That last entry is NOT covered by kubernetes/**: promtool-rules.sh lives
-  # under scripts/, so without it a PR that only edits the script would never run
-  # the check it drives. It is listed as one file rather than scripts/ci/**
-  # because the rest of that directory is CI-agent helpers with nothing to do
-  # with this pipeline, and pulling four images for them is the cost this filter
-  # exists to avoid. Their own tests run in ci-scripts-test.yml on every PR.
-  - event: pull_request
-    path:
-      - "kubernetes/**"
-      - ".woodpecker/**"
-      - "scripts/ci/promtool-rules.sh"
-      - "scripts/ci/yaml-citations.sh"
-
-# THE THIRD DISCOVERY CHECK WENT TO A SCRIPT. kustomize-build and shellcheck
-# both hand-roll the same shape — `find`, a floor guard against empty discovery,
-# then a file-fed `while read`. Two copies was deliberate: the two traps being
-# commented around (BusyBox find lacking `-printf`, and a piped loop running in
-# a subshell where `fail` cannot propagate) are different, and both are worth
-# spelling out where they bite. An earlier revision of this comment said a third
-# instance would be the point to stop copying and factor it into `scripts/ci/`,
-# where it would gain unit tests — and that is what `promtool` below does. Its
-# discovery, floors and guards live in scripts/ci/promtool-rules.sh and are
-# tested by scripts/ci/test-promtool-rules.sh on every PR via
-# ci-scripts-test.yml. A FOURTH inline copy should go the same way; do not add
-# one here.
-#
-# Every step carries `depends_on: []` so all six run in PARALLEL and every
-# failure shows in one run. Woodpecker's default is sequential, which would stop
-# at the first failure and make a contributor fix-and-repush to discover the
-# next — and these six are independent, cheap, offline checks with no ordering
-# between them. Keep it on every step: `depends_on` is per-step, so dropping it
-# from one silently re-serialises that step behind the others.
-steps:
-  # Every kustomization must build. This is the check that catches a resource
-  # entry pointing at a file that does not exist, a bad patch target, or YAML
-  # that is fine in isolation but broken once overlaid — the class of error that
-  # otherwise surfaces as a Flux reconcile failure after merge.
-  #
-  # Verified green across every kustomization in the tree when this was added,
-  # so this gates new breakage rather than pre-existing debt.
-  #
-  # ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml,
-  # alert-monitor.yml and issue-fix.yml, which still sit on an older one. That is
-  # staleness in those four, not a deliberate split here — everything should be on
-  # the newest dated tag. They are not bumped here because none of them are
-  # triggered by this PR, so it would ship untested changes under cover of a green
-  # run that never touched them. Tracked in #461.
-  - name: kustomize-build
-    depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
-    commands:
-      - |
-        set -eu
-        # NOT `find -printf '%h\n'`: that is a GNU findutils extension and this
-        # image ships BusyBox find (v1.37.0), which rejects it outright with
-        # "find: unrecognized: -printf". Because the result feeds a command
-        # substitution used as a word list, the failure is SILENT — empty output,
-        # zero loop iterations, fail=0, step exits green having validated
-        # nothing. That is exactly what this pipeline shipped in its first
-        # revision, and CI reported it as passing.
-        dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort)
-
-        # Assert the discovery actually found something. Without this, any future
-        # breakage of the find/sed above degrades back into a green no-op rather
-        # than a failure — the specific trap that made the first revision of this
-        # step useless while looking healthy.
-        count=$(printf '%s\n' "$dirs" | grep -c . || true)
-        if [ "$count" -lt 50 ]; then
-          echo "FAIL: found only $count kustomizations; floor is 50."
-          echo "      If the tree legitimately shrank, lower this floor in"
-          echo "      .woodpecker/ci-manifests-test.yml — do not delete it."
-          echo "      Otherwise discovery is broken — refusing to report success."
-          exit 1
-        fi
-        echo "discovered $count kustomizations"
-
-        # `while read` fed from a FILE, not a pipe: a pipe would run the loop in
-        # a subshell where `fail` cannot propagate, so a failing build would exit
-        # 0 — the same silent-pass shape this step already shipped once. Reading
-        # a file keeps the loop in the current shell, and read -r handles paths
-        # containing spaces without IFS games.
-        printf '%s\n' "$dirs" > /tmp/dirs
-        fail=0
-        while IFS= read -r d; do
-          [ -n "$d" ] || continue
-          if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
-            echo "ok    $d"
-          else
-            echo "FAIL  $d"
-            sed 's/^/        /' /tmp/err
-            fail=1
-          fi
-        done < /tmp/dirs
-        exit $fail
-
-  # Lint the pipeline definitions themselves. This is the one that would have
-  # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
-  # fails to compile, and exits 0 on mere warnings — so it fails on real breakage
-  # without blocking on pre-existing noise. Both behaviours were verified against
-  # this repo (rc=1 reproducing #443, rc=0 on the tree as it stands).
-  #
-  # An earlier revision of this comment also asserted that several deploy-*.yml
-  # files emit "Additional property timeout is not allowed" warnings. That was
-  # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on current CLIs — review raised the likelihood that the warning was
-  # schema drift in a 7-minor-old CLI rather than real debt in those files, which
-  # is one of the reasons this pin moved to latest stable. It cannot be confirmed
-  # from a green run either, because the loop below sends lint output to /tmp/out
-  # and prints it only on failure, so warnings are invisible when the step passes.
-  # The claim is dropped rather than restated: what this step guarantees is the
-  # exit-code contract above, which is what the gate actually rests on.
-  # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
-  # that, not a separate requirement — the server is bumped by image-automation
-  # within the 3.x range, so both should land on the same release.
-  #
-  # They do not land together, though: the server is a Kubernetes manifest and
-  # is bumped automatically, this pipeline file is not and cannot be (see
-  # infrastructure/image-automation/README.md). So the server moves first and
-  # this trails until someone bumps it by hand — which is what happened at
-  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
-  #
-  # It matters here because the CLI compiles the config against its own schema:
-  # one behind the server can pass a config the server then rejects, or reject
-  # one it accepts, and the gate stops predicting what the server will do —
-  # which is its only purpose.
-  #
-  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no
-  # shell, and the loop below needs `for`, `sed` and `echo` around the binary.
-  # A step using the plain tag fails to start rather than failing a check.
-  #
-  # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
-  # image-automation; this pin cannot, so the two will drift apart.
-  # Image-automation can only patch YAML that is a Kubernetes manifest —
-  # fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
-  # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
-  # pipeline file has neither, so an `$imagepolicy` marker here is silently
-  # ignored. That is a property of the FILE, not of `update.path`: widening the
-  # automation's path to `./` would buy nothing, and would additionally drag the
-  # whole ansible/ and esphome/ trees into the walk. Do not try it — it looks
-  # like the fix and is not.
-  #
-  # So this bump is manual, and drift is caught by the update-audit agent
-  # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
-  # whenever it trails the deployed server. If you bump the server, bump here.
-  - name: woodpecker-lint
-    depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.19.0-alpine@sha256:27a77117e38d7bbdc380f0e90ff3ee4871ab1eefa78b5d9136a2c969fbecd9f4
-    commands:
-      - |
-        set -eu
-        fail=0
-        for f in .woodpecker/*.yml; do
-          if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then
-            echo "ok    $f"
-          else
-            echo "FAIL  $f"
-            sed 's/^/        /' /tmp/out
-            fail=1
-          fi
-        done
-        exit $fail
-
-  # Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts
-  # mounted via configMapGenerator). Severity is pinned to `error` on purpose:
-  # style and info findings would fail the build on existing scripts for no
-  # safety gain. All scripts in the tree are clean at this level today.
-  #
-  # NOTE this does NOT cover shell embedded in manifests (init containers, probe
-  # exec blocks) — shellcheck cannot see inside YAML. That embedded shell is
-  # where several real bugs lived this year, so it remains a gap worth closing
-  # separately rather than one this step quietly implies is covered.
-  - name: shellcheck
-    depends_on: [] # see note above `steps:`
-    image: koalaman/shellcheck-alpine:v0.11.0@sha256:9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002
-    commands:
-      - |
-        set -eu
-        files=$(find kubernetes -name '*.sh' | sort)
-        # Same no-op guard as kustomize-build: if discovery ever returns nothing,
-        # fail loudly instead of passing having checked no files.
-        #
-        # This floor is 1 while kustomize-build's is 50, and the asymmetry is
-        # deliberate rather than an oversight. That floor is loose relative to the
-        # tree it guards, so it catches both total failure and gross partial loss
-        # while tolerating normal churn. The shell-script set is small enough that
-        # any floor tight enough to catch PARTIAL loss would sit inside normal
-        # churn — deleting one script would fail CI spuriously, and the fix a
-        # hurried person reaches for is lowering the floor, which is how a guard
-        # quietly becomes decoration. A floor that has to be edited whenever the
-        # tree changes is not a guard, it is a snapshot with an exit code.
-        #
-        # So this catches the failure that actually happened here (discovery
-        # silently returning nothing, step green having checked no files) and not
-        # partial loss. Worth revisiting only if kubernetes/ grows a lot more
-        # shell, at which point a loose proportionate floor becomes possible.
-        count=$(printf '%s\n' "$files" | grep -c . || true)
-        if [ "$count" -lt 1 ]; then
-          echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
-          exit 1
-        fi
-        echo "discovered $count shell scripts"
-
-        # Same file-fed while-read reasoning as kustomize-build above.
-        printf '%s\n' "$files" > /tmp/files
-        fail=0
-        while IFS= read -r f; do
-          [ -n "$f" ] || continue
-          if shellcheck -S error "$f"; then
-            echo "ok    $f"
-          else
-            echo "FAIL  $f"
-            fail=1
-          fi
-        done < /tmp/files
-        exit $fail
-
-  # Guard the convention that cost a day to diagnose: registry-local images are
-  # pinned by DATED TAG, never by digest. Digests here are garbage-collected once
-  # :latest and the dated tags move past them — and that happens on edits to
-  # .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition
-  # silently orphans the manifest a rotator was pinned to. 11 rotators were dead
-  # this way and would have failed on their first scheduled run months later.
-  #
-  # Scoped to the rotation directory, which is where the convention is documented
-  # and where the failure actually bit.
-  #
-  # The `grep -v kustomization.yaml` is load-bearing, not tidying: that file
-  # documents this very rule, so its comment contains the literal string
-  # `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:`
-  # does not prevent that — verified by running the bare grep, which returns
-  # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
-  # this step fails permanently on its own documentation.
-  - name: no-digest-pins
-    depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
-    commands:
-      - |
-        set -eu
-        hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \
-                 | grep -v kustomization.yaml || true)
-        if [ -n "$hits" ]; then
-          echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):"
-          echo "$hits" | sed 's/^/        /'
-          exit 1
-        fi
-        echo "ok    no digest pins under openbao/rotation/"
-
-  # Machine-check the alert rules: parse every PrometheusRule, run the committed
-  # promtool unit suites, then damage the rules on purpose and require the suites
-  # to notice. Everything it does lives in scripts/ci/promtool-rules.sh; read
-  # that file and kubernetes/apps/monitoring-rules/tests/README.md for the why.
-  #
-  # The gap it closes: the other four steps prove a PrometheusRule assembles into
-  # valid YAML and nothing more — not that an expression parses, not that a label
-  # join matches anything, not that a `for` window does what its comment claims.
-  # On REN-129 the CNPG surge rule was written `for: 2m`, which fires on six of
-  # seven measured nights and would have mailed the owner twice a night at 04:16
-  # and 04:22 local for a backlog that clears itself. Nothing here would have
-  # caught it, and the arithmetic that did catch it was hand-written from scratch
-  # twice, in two runs, and thrown away both times because there was nowhere in
-  # the repo to commit it. REN-152 retired that rule; its `for` margin is now
-  # guarded by mutations.txt:backlog-for-12m.
-  #
-  # Offline, like the rest of this pipeline: no cluster, no registry, no secrets,
-  # no network. promtool reads files and exits.
-  #
-  # IMAGE — prom/prometheus, not a registry-local ci-* image, and pinned to an
-  # exact upstream version rather than a dated tag. The dated-tag convention
-  # guarded by no-digest-pins above is about OUR images, whose digests are
-  # garbage-collected out from under a pin; this is upstream and immutable.
-  #
-  # The version must be 3.x, and that is not a preference. The cluster runs
-  # whatever kube-prometheus-stack >= 72.0.0 ships, which is Prometheus 3, and
-  # alert `for` evaluation is version-sensitive — a 2.x promtool would be
-  # checking a different evaluator than production, so the margins the suites
-  # assert would be claims about the wrong program. promtool-rules.sh asserts the
-  # major version it was handed and fails on 2.x, so a careless bump here is a
-  # red step rather than a quietly wrong check. Keep this at the latest stable
-  # 3.x, per the standing rule that any software uses the latest stable version.
-  #
-  # This image is NOT covered by image-automation (same reason as
-  # woodpecker-lint above: a Woodpecker pipeline file has no apiVersion/kind, so
-  # an $imagepolicy marker here is silently ignored — fluxcd/flux2#2826). The
-  # bump is manual.
-  #
-  # `prom/prometheus` is busybox-based and ships /bin/sh, so it can run a
-  # `commands:` block — unlike the plain woodpecker-cli tag, which cannot. Its
-  # find/grep/awk/sed are BusyBox, which is why promtool-rules.sh avoids
-  # `find -printf` and `grep -r --include`; it was verified against BusyBox
-  # 1.35.0 as well as GNU. The image also declares `USER nobody`, so the script
-  # writes its stripped mirror under mktemp and never into the workspace.
-  - name: promtool
-    depends_on: [] # see note above `steps:`
-    image: prom/prometheus:v3.15.0@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
-    commands:
-      - sh scripts/ci/promtool-rules.sh
-
-  # Comments cite other YAML blocks by anchor name, never by line number
-  # (REN-183; the exact syntax is in CONTRIBUTING.md, not repeated here because
-  # an example written in this comment would itself be checked). Line numbers drift on any edit
-  # above them and nothing noticed: three drift fixes were merged by hand
-  # before this step existed. This fails on a new line-number citation and on
-  # a citation whose file or anchor marker does not exist.
-  # Existence only — it never judges whether the anchored block still says
-  # what the citing comment claims, so ordinary edits cannot turn it red.
-  #
-  # It scans every YAML file in the checkout, but this pipeline only runs on
-  # the paths in `when:` above, so an edit confined to ansible/ or esphome/ is
-  # checked on the next PR that does touch kubernetes/ or .woodpecker/.
-  # Self-test: scripts/ci/test-yaml-citations.sh, in ci-scripts-test.yml.
-  - name: yaml-citations
-    depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
-    commands:
-      - sh scripts/ci/yaml-citations.sh

.woodpecker/ci-scripts-test.yml

@@ -8,57 +8,15 @@
 # against adversarial PR input. Running it here fails the offending PR's checks
 # before it can merge.
 #
-# scripts/ci/test-promtool-rules.sh is the same idea for the alert-rule gate.
-# What it tests is not whether promtool evaluates PromQL — ci-manifests-test.yml
-# does that with the real binary against the real rules — but whether
-# promtool-rules.sh REFUSES TO REPORT SUCCESS when it has checked nothing: zero
-# manifests discovered, zero suites discovered, a `rule_files` path that does not
-# resolve (promtool warns, loads zero rules, and passes every assertion), a
-# mutation the suites no longer catch. Those are the states a green pipeline
-# cannot tell apart from real coverage, so they need a test that asserts the
-# failure. It stubs promtool, so it needs neither the binary nor the heavy image.
-#
-# Both are unfiltered by path on purpose: this pipeline is two steps on one
-# already-cached image, so running it everywhere is free, and a guard whose test
-# only runs when someone remembers to touch the right directory is not a guard.
-#
 # Only in homelab (scripts/ci/ lives here); the file is not deployed to the
-# mirrored repos, so it does not run on their PRs. All tests here are offline (no
-# token, no network, no cluster) — the ci-claude image just provides sh + jq, and
-# it is already pulled by pr-review.yml on the same event, so the runner has it
-# cached.
+# mirrored repos, so it does not run on their PRs. The test is offline (no token,
+# no network, no cluster) — the ci-claude image just provides sh + jq, and it is
+# already pulled by pr-review.yml on the same event, so the runner has it cached.
 when:
   - event: pull_request
 
-# `depends_on: []` on every step so they run in parallel and one failure does not hide
-# the other. Same reasoning as the note above `steps:` in ci-manifests-test.yml.
 steps:
   - name: model-selection-tests
-    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     commands:
       - sh scripts/ci/test-model-selection.sh
-
-  - name: pr-review-mistral-tests
-    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
-    commands:
-      - sh scripts/ci/test-pr-review-mistral.sh
-
-  - name: promtool-rules-guard-tests
-    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
-    commands:
-      - sh scripts/ci/test-promtool-rules.sh
-
-  - name: yaml-citations-guard-tests
-    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
-    commands:
-      - sh scripts/ci/test-yaml-citations.sh
-
-  - name: backup-verify-tests
-    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-backup-verify:2026-10-10
-    commands:
-      - bash kubernetes/infrastructure/velero/backup-verify/tests/run.sh

.woodpecker/deploy-claude-workstation.yml

@@ -6,8 +6,8 @@ when:
       - "ansible/roles/claude-workstation/**"
       - "ansible/plays/install-claude-workstation.yml"
       - ".woodpecker/deploy-claude-workstation.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-claude-workstation"'
+  - event: manual
+    evaluate: 'PIPELINE == "deploy-claude-workstation"'
 
 timeout: 30
 

.woodpecker/deploy-infrastructure.yml

@@ -8,12 +8,12 @@ when:
     path:
       - "opentofu/**"
       - ".woodpecker/deploy-infrastructure.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-infrastructure"'
+  - event: manual
+    evaluate: 'PIPELINE == "deploy-infrastructure"'
 
 steps:
   - name: validate-secrets
-    image: alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
+    image: alpine:3.21
     environment:
       TF_VAR_proxmox_api_url:
         from_secret: proxmox_api_url
@@ -84,7 +84,7 @@ steps:
     image: gitea.renner.uno/gitea_admin/ci-tofu:2026-04-29
     directory: opentofu
     when:
-      - event: deployment
+      - event: manual
         evaluate: 'TOFU_APPLY == "true"'
     environment:
       TF_IN_AUTOMATION: "true"
@@ -128,6 +128,35 @@ steps:
       - printf "Host *\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
       - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
       - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
+      - |
+        # Ensure Talos image exists on all Proxmox nodes via API
+        TALOS_VERSION="v1.12.6"
+        TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
+        TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
+        TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
+        API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
+        for node in jupiter saturn neptune; do
+          EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
+            -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
+          if [ "$EXISTS" -gt 0 ]; then
+            echo "Talos image already exists on $node"
+            continue
+          fi
+          echo "Downloading Talos image to $node..."
+          curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/download-url" \
+            -H "Authorization: PVEAPIToken=$API_TOKEN" \
+            --data-urlencode "url=$TALOS_URL" \
+            --data-urlencode "content=import" \
+            --data-urlencode "filename=$TALOS_FILENAME" \
+            --max-time 600
+          echo "Waiting for download on $node..."
+          for i in $(seq 1 60); do
+            sleep 10
+            EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
+              -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
+            if [ "$EXISTS" -gt 0 ]; then echo "Download complete on $node"; break; fi
+          done
+        done
       - tofu init -upgrade -backend-config="access_key=$AWS_ACCESS_KEY_ID" -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
       - |
         if [ -n "$$TOFU_STATE_RM" ]; then

.woodpecker/deploy-netbird-client.yml

@@ -7,8 +7,8 @@ when:
       - "ansible/plays/install-netbird-client.yml"
       - "ansible/inventory/group_vars/claude_workstation/**"
       - ".woodpecker/deploy-netbird-client.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-client"'
+  - event: manual
+    evaluate: 'PIPELINE == "deploy-netbird-client"'
 
 timeout: 30
 

.woodpecker/deploy-netbird-proxy.yml

@@ -7,24 +7,10 @@ when:
       # install-netbird-proxy.yml also applies the netbird-proxy-watchdog role,
       # so changes to it must redeploy this host too.
       - "ansible/roles/netbird-proxy-watchdog/**"
-      # install-netbird-proxy.yml also applies the edge bouncer.
-      - "ansible/roles/crowdsec-edge-bouncer/**"
       - "ansible/plays/install-netbird-proxy.yml"
-      # The variables that steer those roles, not just the roles themselves.
-      # Without these two, a change that only touches inventory never deploys
-      # and nothing says so: arming the bouncer lives in host_vars, and its
-      # LAPI key lives in group_vars, so both would sit in git looking applied
-      # while the hosts kept running the previous configuration.
-      #
-      # pve_hosts.yml is deliberately NOT listed. It is the single inventory
-      # file for the whole fleet, so listing it would redeploy the proxies for
-      # any unrelated host edit. The cost is that a change to [netbird_proxy]
-      # group membership needs a manual deployment run.
-      - "ansible/inventory/host_vars/proxy-*.yml"
-      - "ansible/inventory/group_vars/netbird_proxy/**"
       - ".woodpecker/deploy-netbird-proxy.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
+  - event: manual
+    evaluate: 'PIPELINE == "deploy-netbird-proxy"'
 
 timeout: 30
 

.woodpecker/deploy-netbird-server.yml

@@ -4,10 +4,13 @@ when:
     branch: master
     path:
       - "ansible/roles/netbird-server/**"
+      # install-netbird-server.yml also applies the netbird-proxy-watchdog role,
+      # so changes to it must redeploy this host too.
+      - "ansible/roles/netbird-proxy-watchdog/**"
       - "ansible/plays/install-netbird-server.yml"
       - ".woodpecker/deploy-netbird-server.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-server"'
+  - event: manual
+    evaluate: 'PIPELINE == "deploy-netbird-server"'
 
 timeout: 30
 

.woodpecker/deploy-uptime-kuma.yml

@@ -1,47 +0,0 @@
-when:
-  # Deploy only on master (merge) + manual — never auto-deploy from a branch.
-  - event: push
-    branch: master
-    path:
-      - "ansible/roles/uptime-kuma/**"
-      - "ansible/plays/install-uptime-kuma.yml"
-      - ".woodpecker/deploy-uptime-kuma.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-uptime-kuma"'
-  - event: cron
-    cron: uptime-kuma-token-renew
-
-timeout: 30
-
-steps:
-  - name: renew-openbao-token
-    image: gitea.renner.uno/gitea_admin/ci-ansible:2026-10-18
-    environment:
-      VAULT_TOKEN:
-        from_secret: openbao_uptime_kuma_token
-    commands:
-      - 'curl -sSf -X POST -H "X-Vault-Token: $VAULT_TOKEN" https://openbao.renner.uno/v1/auth/token/renew-self -o /dev/null'
-
-  - name: deploy-uptime-kuma
-    image: gitea.renner.uno/gitea_admin/ci-ansible:2026-10-18
-    when:
-      - event: [push, deployment]
-    environment:
-      ANSIBLE_FORCE_COLOR: "true"
-      ANSIBLE_HOST_KEY_CHECKING: "false"
-      ANSIBLE_ROLES_PATH: "../roles:./roles"
-      SSH_PRIVATE_KEY:
-        from_secret: ssh_private_key
-      VAULT_PASS:
-        from_secret: vault_pass
-      VAULT_TOKEN:
-        from_secret: openbao_uptime_kuma_token
-    commands:
-      - mkdir -p ~/.ssh
-      - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
-      - chmod 600 ~/.ssh/id_ed25519
-      - printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
-      - cd ansible
-      - echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
-      - export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
-      - ansible-playbook -i inventory/ plays/install-uptime-kuma.yml -v

.woodpecker/e2e-test.yml

@@ -1,10 +1,10 @@
 when:
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e-test"'
+  - event: manual
+    evaluate: 'PIPELINE == "e2e-test"'
 
 steps:
   - name: e2e
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/e2e.yml

@@ -7,8 +7,7 @@
 #                was harmless at daily/weekly cadence — but alert-monitor.yml
 #                fires every 15 minutes, so piggybacking must be opt-in now.
 #                A new cron that SHOULD also run e2e gets its name added here.
-#   - deploy   : `woodpecker-cli pipeline deploy konairius/homelab <n> e2e`, where
-#                <n> is an existing pipeline to promote.
+#   - manual   : `woodpecker pipeline create --branch master --var PIPELINE=e2e k.renner/homelab`
 
 when:
   - event: cron
@@ -17,8 +16,8 @@ when:
     cron: update-audit
   - event: cron
     cron: weekly-update
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e"'
+  - event: manual
+    evaluate: 'PIPELINE == "e2e"'
 
 steps:
   - name: model-selection-tests
@@ -26,12 +25,12 @@ steps:
     # fetch-models.sh response mapping. No cluster, no secrets, no network — the
     # ci-claude image just provides sh + jq. Keeps the flag-injection regression
     # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     commands:
       - sh scripts/ci/test-model-selection.sh
 
   - name: e2e-test
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/hermes-image.yml

@@ -1,29 +0,0 @@
-# Builds ci/images/hermes (nousresearch/hermes-agent plus the agent's CLIs) as
-# gitea.renner.uno/konairius/hermes:build.<unix-ts> and :latest.
-when:
-  - event: push
-    branch: master
-    path:
-      - "ci/images/hermes/**"
-      - ".woodpecker/hermes-image.yml"
-  - event: manual
-
-steps:
-  - name: build
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
-    directory: ci/images/hermes
-    environment:
-      # registry-bot write:package PAT (konairius org secret).
-      GITEA_TOKEN:
-        from_secret: registry_bot_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' registry-bot "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG="build.$(date +%s)"
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/konairius/hermes:$TAG \
-          --destination gitea.renner.uno/konairius/hermes:latest

.woodpecker/install-garage-synology.yml

@@ -0,0 +1,27 @@
+when:
+  - event: manual
+    evaluate: 'PIPELINE == "install-garage-synology"'
+
+steps:
+  - name: run-playbook
+    image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
+    environment:
+      ANSIBLE_FORCE_COLOR: "true"
+      ANSIBLE_HOST_KEY_CHECKING: "false"
+      SSH_PRIVATE_KEY:
+        from_secret: ssh_private_key
+      VAULT_PASS:
+        from_secret: vault_pass
+      PROXMOX_TOKEN_ID:
+        from_secret: proxmox_token_id
+      PROXMOX_TOKEN_SECRET:
+        from_secret: proxmox_token_secret
+    commands:
+      - mkdir -p ~/.ssh
+      - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
+      - chmod 600 ~/.ssh/id_ed25519
+      - 'printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config'
+      - cd ansible
+      - echo "$$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
+      - export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
+      - ansible-playbook -i inventory/ plays/install-garage-synology.yml -v

.woodpecker/issue-fix.yml

@@ -32,15 +32,15 @@
 #
 # Triggering:
 #   - cron   : a cron NAMED exactly "issue-fix":
-#                woodpecker repo cron add --repo konairius/homelab \
+#                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name issue-fix --branch master --schedule '41 * * * *'
-#   - deploy : woodpecker-cli pipeline deploy konairius/homelab <n> issue-fix
-#              where <n> is an existing pipeline to promote.
+#   - manual : woodpecker pipeline create --branch master \
+#                --var PIPELINE=issue-fix k.renner/homelab
 when:
   - event: cron
     cron: issue-fix
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "issue-fix"'
+  - event: manual
+    evaluate: 'PIPELINE == "issue-fix"'
 
 steps:
   - name: fetch-diag
@@ -49,7 +49,7 @@ steps:
     # (the issue-pick logic needs jq, which only the ci-claude image has);
     # minting a 1h read-only token per run is cheap. Degrades to
     # snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key
@@ -62,7 +62,7 @@ steps:
       - bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token
@@ -82,7 +82,7 @@ steps:
         set -eu
         TRUSTED_AUTHORS="k.renner"   # space-separated; the trust anchor (see header)
         mkdir -p /tmp/alert-fix
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
         curl -sfL "$$API/issues?state=open&type=issues&limit=100" \
           | jq '[.[] | select(.title | startswith("[alert] "))] | sort_by(.number)' \
           > /tmp/alert-fix/candidates.json
@@ -163,7 +163,7 @@ steps:
         # run skips this issue, and stop cleanly.
         [ -f /tmp/alert-fix/issue.json ] || exit 0
         set -eu
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
         NUM=$$(jq -r '.number' /tmp/alert-fix/issue.json)
         if git diff --quiet && git diff --cached --quiet \
            && [ -z "$$(git ls-files --others --exclude-standard)" ]; then
@@ -187,7 +187,7 @@ steps:
         [ -f /tmp/alert-fix/issue.json ] || exit 0
         if [ -f /tmp/alert-fix/no-edits ]; then exit 0; fi
         set -eu
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
         NUM=$$(jq -r '.number' /tmp/alert-fix/issue.json)
         BRANCH=$$(cat /tmp/alert-fix/branch)
         TITLE="fix: $$(jq -r '.title' /tmp/alert-fix/issue.json) (#$$NUM)"
@@ -196,7 +196,7 @@ steps:
         printf '\n\nCloses #%s\n' "$$NUM" >> /tmp/alert-fix/summary.md
         git add -A
         git commit -m "$$TITLE"
-        git push "https://$$PUSH_TOKEN@gitea.renner.uno/konairius/homelab.git" HEAD:"$$BRANCH"
+        git push "https://$$PUSH_TOKEN@gitea.renner.uno/k.renner/homelab.git" HEAD:"$$BRANCH"
         jq -n --arg title "$$TITLE" --rawfile body /tmp/alert-fix/summary.md \
           --arg head "$$BRANCH" \
           '{title: $$title, body: $$body, head: $$head, base: "master"}' > /tmp/gitea-pr.json

.woodpecker/pr-review.yml

@@ -1,47 +1,301 @@
-# DISABLED: the review-bot service (review-bot/, deployed from kubernetes/apps/review-bot/) reviews
-# pull requests. This pipeline only starts when triggered by hand and is kept for reference.
+# PR review bot — event-driven. Woodpecker fires this on `pull_request` (which
+# covers BOTH "PR opened" and "new commits pushed to an open PR"), so a review
+# is produced the moment a PR appears or is updated — no polling, no cron.
 #
-# PR review bot — per-repo STUB. All logic lives centrally in
-# konairius/homelab: ci/agents/pr-review/run.sh (plus prompt.md,
-# settings.json and scripts/ci/). This file is deployed byte-identical into every
-# reviewed repo (konairius/homelab, shelly-manager, authentik-samba-bridge,
-# homeassistant) and holds NO logic, so the copies cannot drift apart: change the
-# review in homelab and every repo picks it up on its next PR event. Only edit this
-# stub if the trigger, the image or the secrets change — then change all copies in
-# one go.
+# This file is deployed UNCHANGED into every repo the fleet wants reviewed
+# (k.renner/homelab, shelly-manager, authentik-samba-bridge, homeassistant).
+# Woodpecker events are per-repo, so each repo triggers a review of ITS OWN PR
+# only; the triggering PR is identified from the CI event (CI_REPO +
+# CI_COMMIT_PULL_REQUEST) instead of the old hourly cross-repo discovery loop.
+# Because it runs on the PR event, Woodpecker's built-in clone hands the step
+# THIS repo already checked out at the PR head — so, unlike the old cron model,
+# there is no manual clone of the target repo (the source of an earlier token
+# leak); the agents read the checkout in the workspace directly.
 #
-# Triggers on pull_request: PR opened AND new commits pushed to an open PR.
-# Approves only when the review found nothing red/yellow (see run.sh); humans merge.
+# Prompts: the two agent prompts live once, in k.renner/homelab under
+# ci/agents/pr-review/. The homelab checkout already has them; the other repos
+# don't, so the step falls back to a shallow clone of public homelab@master.
+# One source of truth for the review rubric across all repos.
 #
-# The central code is cloned from public homelab@master on every run, never taken
-# from the PR checkout, so a PR cannot alter the rubric or helper scripts it is
-# reviewed with.
+# Model selection: no model id — and no model family — is baked into this file.
+# scripts/ci/fetch-models.sh pulls the LIVE catalog (Anthropic /v1/models, via
+# CLAUDE_CODE_OAUTH_TOKEN), and a triage agent reads the PR diff + stats and
+# picks whichever currently-available model best fits the review, judged on the
+# catalog's own capability metadata. A model released tomorrow is eligible the
+# day it appears. The shell validates the pick against the catalog snapshot taken
+# BEFORE the triage runs; if the fetch fails, the pick is rejected, or the review
+# is complex/security-sensitive (where the injectable pick is deliberately
+# ignored), it leaves the model unset and the review runs on the CLI's default
+# model. A wrong-way miss only over/under-spends one review.
 #
-# Image: ci-claude:latest — the refresh pipeline keeps it on current Claude Code,
-# and a pinned tag would be the one thing in this file that diverges per repo.
+# Posture: posted as a Gitea PR review, event COMMENT normally or
+# REQUEST_CHANGES on a serious defect. It NEVER approves — a bot must not
+# satisfy branch protection. Humans still merge.
 #
-# Secrets (both must be granted the pull_request event):
-#   mistral_api_key         -> MISTRAL_API_KEY (Mistral Conversations API)
-#   review_bot_token        -> REVIEW_TOKEN (review-bot PAT, write:repository)
-# Claude CLI path disabled: claude_code_oauth_token -> CLAUDE_CODE_OAUTH_TOKEN
+# Dedup (guards pipeline retries/restarts, not cron re-runs anymore): the PR is
+# skipped when review-bot already left a review carrying the marker
+# "<!-- pr-review:sha=<head_sha> -->" for the PR's CURRENT head commit. A pushed
+# commit changes the head sha and fires a fresh pull_request event -> fresh
+# review.
 #
-# (Note: keep literal dollar-brace out of THIS file, even in comments — Woodpecker
-# runs its variable substitution over the whole YAML text.)
+# Tokens:
+#   CLAUDE_CODE_OAUTH_TOKEN (claude_code_oauth_token) — Claude subscription
+#     OAuth, how the `claude` CLI authenticates (same as the other agents).
+#   REVIEW_TOKEN (review_bot_token) — a dedicated `review-bot` Gitea account's
+#     PAT, scope write:repository. Reads the PR + diff and POSTs the review, all
+#     over the Gitea API (ephemeral curl Authorization headers — the token is
+#     never written to disk; the repo is NOT cloned). review-bot must be a
+#     collaborator (read) on every reviewed repo.
+#   Both secrets MUST be granted the `pull_request` event, or Woodpecker will
+#     withhold them from this pipeline. Because homelab is PUBLIC, it is also set
+#     to require approval for forked-PR pipelines, so an untrusted fork cannot
+#     rewrite this file to exfiltrate the tokens.
+#
+# Triggering: pull_request only — automatic on PR open + push-to-open-PR. To
+# re-review a PR, push a commit (the new head sha clears the dedup marker below
+# and earns a fresh review). There is deliberately no manual PIPELINE trigger:
+# Woodpecker exposes a --var only to YAML variable substitution and `when`
+# filters, NOT to a step's shell environment, so a --var PR=<n> re-review would
+# silently no-op (the step would find no PR number and exit green).
+# (Note: keep literal dollar-brace out of THIS file entirely, even in comments —
+# Woodpecker runs its variable substitution over the whole YAML text, comments
+# included, and an unparseable one errors the pipeline before any step runs.)
 when:
-  - event: manual
+  - event: pull_request
 
 steps:
   - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:latest
-    pull: true
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     environment:
-      # CLAUDE_CODE_OAUTH_TOKEN:
-      #   from_secret: claude_code_oauth_token
-      MISTRAL_API_KEY:
-        from_secret: mistral_api_key
+      CLAUDE_CODE_OAUTH_TOKEN:
+        from_secret: claude_code_oauth_token
       REVIEW_TOKEN:
         from_secret: review_bot_token
     commands:
-      - rm -rf /tmp/pr-review-central
-      - git clone --quiet --depth 1 https://gitea.renner.uno/konairius/homelab /tmp/pr-review-central
-      - sh /tmp/pr-review-central/ci/agents/pr-review/run.sh
+      - |
+        # ---- Resolve the single triggering PR --------------------------------
+        # Woodpecker hands us the repo (CI_REPO, canonical owner/repo) and the PR
+        # number (CI_COMMIT_PULL_REQUEST, a pull_request-event built-in). Build
+        # /tmp/pr-review/worklist.jsonl with 0 or 1 entries; an empty file makes
+        # the next block a no-op (the agent never starts).
+        set -eu
+        API="https://gitea.renner.uno/api/v1"
+        AUTH="Authorization: token $$REVIEW_TOKEN"
+        SLUG="$$CI_REPO"
+        NUM="$${CI_COMMIT_PULL_REQUEST:-}"
+        REVIEW_BOT_LOGIN="review-bot"
+        mkdir -p /tmp/pr-review
+        : > /tmp/pr-review/worklist.jsonl
+
+        if [ -z "$$NUM" ]; then
+          echo "No PR number in CI env (event=$${CI_PIPELINE_EVENT:-?}) — nothing to review."
+          exit 0
+        fi
+        echo "Triggered for $$SLUG#$$NUM (event $${CI_PIPELINE_EVENT:-?})."
+
+        if ! curl -sf -H "$$AUTH" "$$API/repos/$$SLUG/pulls/$$NUM" > /tmp/pr-review/pr.json; then
+          echo "Cannot fetch $$SLUG#$$NUM (review-bot not a collaborator / PR gone?) — skipping."
+          exit 0
+        fi
+        STATE=$$(jq -r '.state'        /tmp/pr-review/pr.json)
+        DRAFT=$$(jq -r '.draft // false' /tmp/pr-review/pr.json)
+        TITLE=$$(jq -r '.title'        /tmp/pr-review/pr.json)
+        HEAD=$$(jq -r '.head.sha'      /tmp/pr-review/pr.json)
+        SHORT=$$(printf '%s' "$$HEAD" | cut -c1-8)
+
+        # Skip closed/merged, drafts, and WIP-titled PRs — not ready for review.
+        case "$$STATE" in open) ;; *) echo "#$$NUM is $$STATE — skip."; exit 0;; esac
+        case "$$DRAFT" in true) echo "#$$NUM draft — skip."; exit 0;; esac
+        case "$$TITLE" in [Ww][Ii][Pp]:*|[Ww][Ii][Pp]\ *) echo "#$$NUM WIP — skip."; exit 0;; esac
+
+        # Dedup: already reviewed at THIS head sha? (guards re-runs / restarts)
+        REVIEWED=$$(curl -sf -H "$$AUTH" "$$API/repos/$$SLUG/pulls/$$NUM/reviews?limit=50" \
+          | jq -r --arg u "$$REVIEW_BOT_LOGIN" --arg m "pr-review:sha=$$HEAD" \
+              '[.[] | select(.user.login == $$u) | select(.body | contains($$m))] | length' \
+          || echo 0)
+        if [ "$${REVIEWED:-0}" -gt 0 ]; then
+          echo "#$$NUM already reviewed at $$SHORT — skip."
+          exit 0
+        fi
+
+        jq -c --arg slug "$$SLUG" \
+          '{slug: $$slug, number: .number, head: .head.sha, base: .base.sha,
+            title: .title, author: .user.login}' \
+          /tmp/pr-review/pr.json >> /tmp/pr-review/worklist.jsonl
+        echo "#$$NUM enqueued (head $$SHORT)."
+      - |
+        # ---- Clone, pick model, review, post ---------------------------------
+        set -eu
+        [ -s /tmp/pr-review/worklist.jsonl ] || { echo "Nothing to review."; exit 0; }
+        API="https://gitea.renner.uno/api/v1"
+        AUTH="Authorization: token $$REVIEW_TOKEN"
+        WS="$$PWD"                     # workspace = this repo, checked out at the PR head by Woodpecker
+        MAX_DIFF_LINES=20000
+
+        # Agent prompts: use the local homelab checkout if present, else clone
+        # public homelab@master for them (one source of truth for the rubric).
+        PROMPTS="$$WS/ci/agents/pr-review"
+        HOMELAB="$$WS"                 # repo root holding ci/ AND scripts/ci/
+        # NB: for a homelab PR, HOMELAB is the PR head checkout, so the
+        # fetch-models.sh run below executes branch-supplied shell. It is wrapped
+        # in `env -u REVIEW_TOKEN`, so the only credential it can see is
+        # CLAUDE_CODE_OAUTH_TOKEN. This is not a new exposure regardless: on
+        # pull_request Woodpecker already takes .woodpecker/*.yml itself from the
+        # PR branch, so a branch can run arbitrary commands in this step anyway.
+        if [ ! -f "$$PROMPTS/prompt.md" ]; then
+          rm -rf /tmp/pr-review/homelab
+          git clone --quiet --depth 1 https://gitea.renner.uno/k.renner/homelab \
+            /tmp/pr-review/homelab
+          PROMPTS=/tmp/pr-review/homelab/ci/agents/pr-review
+          HOMELAB=/tmp/pr-review/homelab
+        fi
+        echo "Using prompts from $$PROMPTS"
+
+        gitea_post() {                 # method url datafile ; echoes+checks HTTP
+          code=$$(curl -s -o /tmp/pr-review/resp.json -w '%{http_code}' \
+            -X "$$1" -H "$$AUTH" -H "Content-Type: application/json" \
+            --data @"$$3" "$$2")
+          echo "$$1 $$2 -> HTTP $$code"
+          if [ "$$code" -lt 200 ] || [ "$$code" -ge 300 ]; then
+            echo "Gitea API error body:"; cat /tmp/pr-review/resp.json; echo; return 1
+          fi
+        }
+
+        while IFS= read -r ITEM; do
+          SLUG=$$(printf '%s' "$$ITEM"  | jq -r '.slug')
+          NUM=$$(printf '%s' "$$ITEM"   | jq -r '.number')
+          HEAD=$$(printf '%s' "$$ITEM"  | jq -r '.head')
+          TITLE=$$(printf '%s' "$$ITEM" | jq -r '.title')
+          SHORT=$$(printf '%s' "$$HEAD" | cut -c1-8)
+          echo "=== Reviewing $$SLUG#$$NUM ($$SHORT) ==="
+          D="/tmp/pr-review/$$NUM"
+          rm -rf "$$D"; mkdir -p "$$D"
+
+          # PR metadata + unified diff (authenticated; private repos need it).
+          curl -sf -H "$$AUTH" "$$API/repos/$$SLUG/pulls/$$NUM" > "$$D/pr.json"
+          curl -sf -H "$$AUTH" "$$API/repos/$$SLUG/pulls/$$NUM.diff" > "$$D/diff.full.patch" \
+            || { echo "Could not fetch diff for $$SLUG#$$NUM — skipping."; continue; }
+          FILES=$$(grep -c '^diff --git ' "$$D/diff.full.patch" || true)
+          ADDED=$$(grep -c '^+[^+]' "$$D/diff.full.patch" || true)
+          DELETED=$$(grep -c '^-[^-]' "$$D/diff.full.patch" || true)
+          TOTAL=$$(wc -l < "$$D/diff.full.patch")
+          if [ "$$TOTAL" -gt "$$MAX_DIFF_LINES" ]; then
+            head -n "$$MAX_DIFF_LINES" "$$D/diff.full.patch" > "$$D/diff.patch"
+            printf '\n... [diff truncated at %s of %s lines] ...\n' \
+              "$$MAX_DIFF_LINES" "$$TOTAL" >> "$$D/diff.patch"
+          else
+            cp "$$D/diff.full.patch" "$$D/diff.patch"
+          fi
+          printf 'repo: %s\nPR: #%s\nfiles_changed: %s\nlines_added: %s\nlines_deleted: %s\ndiff_total_lines: %s\n' \
+            "$$SLUG" "$$NUM" "$$FILES" "$$ADDED" "$$DELETED" "$$TOTAL" > "$$D/stats.txt"
+
+          # Cheaper inputs for the TRIAGE agent (which now runs on a capable
+          # default model): the COMPLETE changed-file list, plus a diff capped far
+          # smaller than the review's. Triage only gauges size/shape/sensitivity,
+          # so it never needs the full (up to 20k-line) review diff — this keeps
+          # "triage is cheap" true now that its model is no longer pinned to Haiku.
+          grep '^diff --git ' "$$D/diff.full.patch" | sed 's|^diff --git a/||; s| b/.*$$||' \
+            | sort -u > "$$D/files.txt"
+          head -n 4000 "$$D/diff.patch" > "$$D/diff.triage.patch"
+
+          # No manual clone of the target repo: this runs on a pull_request
+          # event, so Woodpecker's built-in clone already checked THIS repo out
+          # at the PR head in $$WS — the agents (which cd into $$WS) read it
+          # directly for context. The old cron model had to clone each target
+          # repo by hand (its workspace was homelab@master), and that clone
+          # embedded REVIEW_TOKEN in the URL, persisting a write-scoped PAT in
+          # work/target/.git/config — readable by the review agent whose report
+          # is posted publicly. Per-repo pull_request triggering makes the manual
+          # clone redundant AND keeps the token off disk (it now lives only in
+          # the ephemeral curl Authorization headers used for the API calls).
+
+          # Inputs the agents read live at ./.pr-review (symlink into $$D).
+          rm -f "$$WS/.pr-review"; ln -s "$$D" "$$WS/.pr-review"
+
+          # --- Triage: an agent picks the model from the LIVE catalog, so no
+          #     model id and no model family is baked into this file.
+          #     fetch-models.sh asks the Anthropic /v1/models list (via the same
+          #     CLAUDE_CODE_OAUTH_TOKEN) and we hand the triage the result at
+          #     ./.pr-review/models.txt. If the fetch fails the file is empty and
+          #     MODEL stays empty below, so the review runs on the CLI default.
+          if MODELS_TABLE=$$(env -u REVIEW_TOKEN sh "$$HOMELAB/scripts/ci/fetch-models.sh" --table); then
+            printf '%s\n' "$$MODELS_TABLE" > "$$D/models.txt"
+            echo "Live model catalog ($$(wc -l < "$$D/models.txt") models) written for triage."
+          else
+            : > "$$D/models.txt"
+            echo "Model catalog fetch failed — triage/review will use the CLI default model."
+          fi
+          # env -u REVIEW_TOKEN: the agents never call Gitea (the shell does, via
+          # $$AUTH), so the write:repository PAT must not sit in the agent's env
+          # where a prompt-injected diff could read /proc/self/environ. Defence in
+          # depth alongside the settings.json Read-sandbox. Triage itself runs on
+          # the CLI default model (no --model) — we don't name a family even to
+          # bootstrap the pick.
+          cat "$$PROMPTS/triage-prompt.md" | env -u REVIEW_TOKEN claude \
+            --permission-mode acceptEdits \
+            --allowedTools "Read,Glob,Grep,Write" \
+            --settings "$$PROMPTS/settings.json" \
+            -p || echo "(triage agent exited non-zero)"
+          # Use the triage's pick only if it is a currently-available id;
+          # otherwise (hallucinated id, or empty catalog) leave MODEL empty so
+          # the review runs on the CLI default — a run never dies on selection.
+          MODEL=$$(jq -r '.model // empty' "$$D/triage.json" 2>/dev/null || true)
+          CPLX=$$(jq -r '.complexity // "unknown"' "$$D/triage.json" 2>/dev/null || echo unknown)
+          # triage.json is UNTRUSTED (the triage reads the attacker-controlled
+          # diff and its Write is not path-scoped). valid-model.sh is the single,
+          # unit-tested gate (scripts/ci/test-model-selection.sh): it echoes the id
+          # only if it is a safe [A-Za-z0-9._-] token present in the catalog, else
+          # nothing — so a value like "claude-x\n--dangerously-skip-permissions"
+          # can never reach the unquoted --model expansion below. Empty -> default.
+          # CRITICAL: validate against $$MODELS_TABLE — the catalog captured in a
+          # shell variable BEFORE the triage ran — NOT the on-disk ./.pr-review/
+          # models.txt, which the triage's unscoped Write could overwrite to smuggle
+          # an attacker-chosen token past the membership check (the triage reads the
+          # file, but only $$MODELS_TABLE is trusted). This mirrors pick-model.sh,
+          # which validates its pick against its in-memory table for the same reason.
+          # env -u REVIEW_TOKEN: like fetch-models.sh above, this runs a script from
+          # $$HOMELAB (branch-supplied for a homelab PR) — it must not see the PAT.
+          # `|| true`: keep the "a run never dies on selection" invariant even if
+          # the script is absent (e.g. a PR that moves scripts/ci/).
+          MODEL=$$(printf '%s\n' "$$MODELS_TABLE" | env -u REVIEW_TOKEN sh "$$HOMELAB/scripts/ci/valid-model.sh" "$$MODEL" || true)
+          [ -n "$$MODEL" ] || echo "Triage model rejected or not in live catalog — using the CLI default model."
+          # Defence in depth: for a complex / security-sensitive review, do NOT
+          # trust the triage's pick (it read the attacker-controlled diff and could
+          # be injected into naming the weakest catalog entry to get a shallow
+          # review). Fall back to the CLI default — the strongest configured model.
+          if [ "$$CPLX" = complex ] && [ -n "$$MODEL" ]; then
+            echo "complex review — ignoring triage pick '$$MODEL', using the CLI default (strongest) model."
+            MODEL=
+          fi
+          if [ -n "$$MODEL" ]; then echo "Triage: complexity=$$CPLX -> model=$$MODEL"; else echo "Triage: complexity=$$CPLX -> CLI default model"; fi
+
+          # --- Review with the chosen model ---
+          ( cd "$$WS" && cat "$$PROMPTS/prompt.md" | env -u REVIEW_TOKEN claude $${MODEL:+--model $$MODEL} \
+              --permission-mode acceptEdits \
+              --allowedTools "Read,Glob,Grep,Write" \
+              --settings "$$PROMPTS/settings.json" \
+              -p ) || echo "(review agent exited non-zero)"
+
+          if [ ! -s "$$D/report.md" ]; then
+            echo "Agent wrote no report for $$SLUG#$$NUM — skipping post."
+            continue
+          fi
+          EVENT=$$(jq -r '.event // "COMMENT"' "$$D/verdict.json" 2>/dev/null || echo COMMENT)
+          case "$$EVENT" in REQUEST_CHANGES|COMMENT) ;; *) EVENT=COMMENT;; esac   # never APPROVE
+
+          # Assemble the review body: header + report + marker footer.
+          {
+            printf '🤖 **Automated review** — model `%s` (complexity: %s), commit `%s`.\n\n' \
+              "$${MODEL:-CLI default}" "$$CPLX" "$${HEAD}"
+            printf '> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.\n\n---\n\n'
+            cat "$$D/report.md"
+            printf '\n\n<!-- pr-review:sha=%s -->\n' "$$HEAD"
+          } > "$$D/body.md"
+          jq -n --rawfile b "$$D/body.md" --arg e "$$EVENT" --arg c "$$HEAD" \
+            '{body: $$b, event: $$e, commit_id: $$c}' > "$$D/review.json"
+          gitea_post POST "$$API/repos/$$SLUG/pulls/$$NUM/reviews" "$$D/review.json" \
+            && echo "Posted $$EVENT review on $$SLUG#$$NUM." \
+            || echo "FAILED to post review on $$SLUG#$$NUM (continuing)."
+          rm -f "$$WS/.pr-review"
+        done < /tmp/pr-review/worklist.jsonl
+        echo "PR review run complete."

.woodpecker/review-bot-test.yml

@@ -1,13 +0,0 @@
-# Tests for the review-bot Go service (review-bot/). Offline: the tests run against fake Gitea and
-# Mistral servers, so the step needs no secret and no network beyond the module cache.
-when:
-  - event: pull_request
-    path: ["review-bot/**", ".woodpecker/review-bot-test.yml"]
-
-steps:
-  - name: go-test
-    image: golang:1.27.2-bookworm@sha256:5cf287a799e6b94384bad13d16b14904c531f51ba65792237e122ce42b392f61
-    commands:
-      - test -z "$(gofmt -l review-bot)"
-      - go -C review-bot vet ./...
-      - go -C review-bot test -race ./...

.woodpecker/security-scan.yml

@@ -1,29 +1,17 @@
-# Trivy scan of the IaC tree and every image referenced from ansible/, with an
-# agent pass that opens a PR for CRITICAL/HIGH findings.
-#
-# Triggers:
-#   - cron       : the named `daily-security-scan` cron, matched by NAME.
-#                  Crons cannot carry variables — `woodpecker-cli repo cron add`
-#                  has no --var — so a variable gate can never match a cron.
-#   - deployment : woodpecker-cli pipeline deploy konairius/homelab <n> security-scan
-#                  The target rides in pipeline metadata as
-#                  CI_PIPELINE_DEPLOY_TARGET, which the workflow-root filter
-#                  does see, unlike manual-run variables on Woodpecker 3.18
-#                  (#582).
 when:
   - event: cron
-    cron: daily-security-scan
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "security-scan"'
+    evaluate: 'PIPELINE == "security-scan"'
+  - event: manual
+    evaluate: 'PIPELINE == "security-scan"'
 
 steps:
   - name: iac-scan
-    image: aquasec/trivy:latest@sha256:2416d1f74045b57274df1a340eee6b510cacd73c66a6ccb8da092b117eb7d222
+    image: aquasec/trivy:latest
     commands:
       - trivy fs --format json -o trivy-iac.json --severity CRITICAL,HIGH --ignore-unfixed .
 
   - name: image-scans
-    image: aquasec/trivy:latest@sha256:2416d1f74045b57274df1a340eee6b510cacd73c66a6ccb8da092b117eb7d222
+    image: aquasec/trivy:latest
     commands:
       - trivy image --download-db-only
       - |
@@ -40,28 +28,12 @@ steps:
         cat /tmp/images.txt
         echo "---"
         i=0
-        failed=0
-        failed_names=""
         while IFS= read -r img; do
           i=$$((i+1))
           out="trivy-image-$$i.json"
           echo "=== [$$i] $$img -> $$out ==="
-          # Keep going on a bad image so one unpullable reference cannot cost
-          # the scan of every image after it, but record that it happened.
-          if ! trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img"; then
-            failed=$$((failed+1))
-            failed_names="$$failed_names $$img"
-          fi
+          trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img" || true
         done < /tmp/images.txt
-        echo "--- scanned $$((i - failed)) of $$i images ---"
-        if [ "$$failed" -gt 0 ]; then
-          echo "NOT SCANNED:$$failed_names"
-          exit 1
-        fi
-    # Non-zero above turns this step red without failing the workflow, so the
-    # fix agent still runs on what was scanned. Coverage is reported, not
-    # enforced: an incomplete scan is visible in the step status instead of
-    # being indistinguishable from a clean one.
     failure: ignore
 
   - name: security-fix-agent
@@ -70,7 +42,7 @@ steps:
     # this step is skipped — otherwise we'd fork-bomb PRs.
     when:
       - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token
@@ -147,7 +119,7 @@ steps:
         TITLE="security: agent-proposed fixes ($$(date +%Y-%m-%d))"
         git add -A
         git commit -m "$$TITLE"
-        git push "https://$$GITEA_TOKEN@gitea.renner.uno/konairius/homelab.git" HEAD:"$$BRANCH"
+        git push "https://$$GITEA_TOKEN@gitea.renner.uno/k.renner/homelab.git" HEAD:"$$BRANCH"
         jq -n \
           --arg title "$$TITLE" \
           --arg body  "$$BODY"  \
@@ -157,5 +129,5 @@ steps:
             -H "Authorization: token $$GITEA_TOKEN" \
             -H "Content-Type: application/json" \
             --data @- \
-            https://gitea.renner.uno/api/v1/repos/konairius/homelab/pulls \
+            https://gitea.renner.uno/api/v1/repos/k.renner/homelab/pulls \
         | jq '{number, html_url, title}'

.woodpecker/update-all.yml

@@ -1,19 +1,13 @@
 when:
-  # Deploy only on master (merge) + manual — never auto-deploy from a branch.
-  - event: push
-    branch: master
-    path:
-      - "ansible/roles/mqtt-bridge/**"
-      - "ansible/plays/edge-mqtt-bridge.yml"
-      - "ansible/inventory/group_vars/edge_host/**"
-      - ".woodpecker/deploy-edge-mqtt-bridge.yml"
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-edge-mqtt-bridge"'
+  - event: cron
+    evaluate: 'PIPELINE == "update-all"'
+  - event: manual
+    evaluate: 'PIPELINE == "update-all"'
 
-timeout: 30
+timeout: 120
 
 steps:
-  - name: deploy-edge-mqtt-bridge
+  - name: update-all
     image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
     environment:
       ANSIBLE_FORCE_COLOR: "true"
@@ -23,6 +17,10 @@ steps:
         from_secret: ssh_private_key
       VAULT_PASS:
         from_secret: vault_pass
+      PROXMOX_TOKEN_ID:
+        from_secret: proxmox_token_id
+      PROXMOX_TOKEN_SECRET:
+        from_secret: proxmox_token_secret
     commands:
       - mkdir -p ~/.ssh
       - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
@@ -31,6 +29,4 @@ steps:
       - cd ansible
       - echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
       - export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
-      # Local MQTT bridge on the edge hosts only (REN-1307); restarts only the
-      # local mosquitto on edge-1, never the cluster broker.
-      - ansible-playbook -i inventory/ plays/edge-mqtt-bridge.yml -v
+      - ansible-playbook -i inventory/ plays/update.yml -v

.woodpecker/update-audit.yml

@@ -10,14 +10,14 @@
 # report; the pipeline posts it to a single persistent Gitea issue.
 #
 # Triggering:
-#   - cron   : a cron NAMED exactly "update-audit". Crons carry no variables —
-#              `repo cron add` takes name, branch, schedule and enabled — so the
-#              name filter is the only thing that can select a workflow for a
-#              scheduled run:
-#                woodpecker repo cron add --repo konairius/homelab \
+#   - cron   : a cron NAMED exactly "update-audit". Unlike the other pipelines
+#              here (which gate on an injected `PIPELINE` var set at cron-create
+#              time in the UI), this uses Woodpecker's documented `cron:` name
+#              filter so the schedule can be created entirely from the CLI:
+#                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name update-audit --branch master --schedule '@weekly'
-#   - deploy : woodpecker-cli pipeline deploy konairius/homelab <n> update-audit
-#              where <n> is an existing pipeline to promote.
+#   - manual : woodpecker pipeline create --branch master \
+#                --var PIPELINE=update-audit k.renner/homelab
 #
 # NOTE: e2e.yml runs on an explicit list of cron names that includes this one,
 # so a weekly e2e run piggybacks on the audit (it was a bare `event: cron`
@@ -25,12 +25,12 @@
 when:
   - event: cron
     cron: update-audit
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "update-audit"'
+  - event: manual
+    evaluate: 'PIPELINE == "update-audit"'
 
 steps:
   - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token
@@ -111,7 +111,7 @@ steps:
         set -eu
         TITLE="Infrastructure update audit"
         STAMP=$$(date -u +%Y-%m-%dT%H:%M:%SZ)
-        API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
 
         # write <method> <url> <payload-file>  -> prints HTTP code, fails non-2xx
         gitea_write() {

CONTRIBUTING.md

@@ -1,7 +0,0 @@
-# Contributing
-
-## Citing other YAML from a comment
-
-- Mark the block you cite with a comment line `# anchor: <name>` (kebab-case, unique in that file). If the block is inside a `|` block scalar, put the marker just above the key, never inside the scalar.
-- Cite it as `path.yaml#<name>`, with the path relative to the citing file's directory or to the repo root. Never cite `path.yaml:<line>`, because line numbers go stale on any edit above them.
-- `scripts/ci/yaml-citations.sh` enforces this in `ci-manifests-test.yml`: it fails on line-number citations and on citations whose file or anchor does not exist.

README.md

@@ -1,49 +0,0 @@
-# homelab
-
-GitOps source for the Renner homelab cluster. Flux applies `master` to
-production, so every merge is a production change. Changes go in by pull
-request only; see [CONTRIBUTING.md](CONTRIBUTING.md) for YAML conventions.
-
-## Change rules that protect Home Assistant
-
-Owner decision (REN-240, REN-243). Home Assistant runs the house, so it
-gets two rules on top of normal review.
-
-### 1. Disruptive work runs only 02:00–05:00 Europe/Berlin
-
-Planned work that can restart Home Assistant or disrupt the cluster is
-merged and applied only between **02:00 and 05:00 Europe/Berlin**. That
-covers:
-
-- node drain, node reboot, Talos upgrade
-- CNI (Cilium, Multus) and cluster-wide network policy
-- storage (Rook/Ceph, CSI, snapshot controller, CNPG operator)
-- Flux core (`kubernetes/clusters/*/flux-system/`, Flux controllers)
-- Authentik
-- OpenBao, including secret rotation that restarts consumers
-- anything in `kubernetes/apps/home-assistant/`
-
-A PR in this list states in its description that it waits for the window.
-Flux applies on merge, so "wait for the window" means **merge** in the
-window, not just apply. If unsure whether a change is disruptive, treat it
-as disruptive.
-
-Not covered: Home Assistant image bumps by Flux image automation. They stay
-fully automatic by owner decision. If one breaks Home Assistant, roll it
-back with [RUNBOOK-home-assistant-image-rollback.md](kubernetes/RUNBOOK-home-assistant-image-rollback.md).
-An incident rollback is not planned work; the owner decides its timing.
-
-### 2. HA-impact review before merge
-
-Every PR that touches `kubernetes/apps/home-assistant/`, or makes a
-cluster-wide disruptive change from the list above, needs an **HA-impact
-review** before merge: a named reviewer states in the PR what Home
-Assistant will notice (restart, lost integration, lost history, none) and
-approves.
-
-- Reviewer: Hestia, the Home Assistant maintainer.
-- Until Hestia exists: the owner does the HA-impact review.
-
-This is in addition to the normal review and merge by Argus. Argus does
-not merge such a PR without the HA-impact approval.
-

ansible/inventory/api.proxmox.yaml

@@ -0,0 +1,22 @@
+plugin: community.proxmox.proxmox
+user: root@pam
+# token_id: ansible
+# token_secret: provided via PROXMOX_TOKEN_SECRET env var
+url: 'https://jupiter.renner.uno:8006'
+validate_certs: true
+want_facts: true
+want_nodes: false
+keyed_groups:
+    # proxmox_tags_parsed is an example of a fact only returned when 'want_facts=true'
+  - key: proxmox_tags_parsed
+    separator: "_"
+    prefix: proxmox
+  - key: proxmox_status
+    prefix: status
+compose:
+  ansible_host: >-
+    (proxmox_lxc_interfaces | default([]) | selectattr('name', 'equalto', 'eth0') | map(attribute='inet') | list | first | default('') | split('/') | first)
+    if proxmox_vmtype == 'lxc' else
+    (proxmox_agent_interfaces | default([]) | selectattr('name', 'equalto', 'eth0') | map(attribute='ip-addresses') | list | first | list | first | default('') | split('/') | first)
+  ansible_user: "'root' if proxmox_vmtype == 'lxc' else 'ladmin'"
+

ansible/inventory/group_vars/all/netbird-rotatable.yml

@@ -1,13 +0,0 @@
-$ANSIBLE_VAULT;1.1;AES256
-32373134633930343261336362373335623735656634313034393731393661643361346634623439
-3561363963333533313264643036663731656465343162330a333964383462373934303766323533
-63613065313063626261383236396363656565623538666565646661353766643133306334396561
-3961376234623236620a616339326238366538333666316538396465666365373834313064653963
-38623233616633383435373836613438333763343932636239386263306238376137643531373161
-35386437313465643437343832623935353238316565366537646530323233363837656631383962
-32653332333162623966366163633732356637323461613966313263313066643966373335313162
-30306362346561633566316166313164323635356363643034663933363464663562343136646137
-66643536363734316533663032643637356136363237376365613330633961616665323430633135
-36323733633464306330666232663433616533373466633939353061663136666665323432373666
-36393930333435333238306238373931393232393861333334643139306138653336633534626362
-61323038303838396438

ansible/inventory/group_vars/all/vault.yml

@@ -1,29 +1,201 @@
 $ANSIBLE_VAULT;1.1;AES256
-30353730633631376539373334376531353263396339313464373538396231616166653738383161
-6438656132616365646463653464396236313437323537620a363632643061333036353537626461
-34393065613836343536383565303066326133363937386263653961663666643430663139653961
-6462363235343063350a636165376232366636363233376637636231363839343738346134623562
-63663839326435303964313730336436396365343163343366613966323938656263303166326435
-65303931393233623064343763633161353161643635396365316362333137353732346366643162
-36386132386638623333623839373161386330353635346331393735366662393935353264646439
-36386539393432373765396562613033656339646532323864366230393638313161383363636636
-32353266376563393239393734316461313566386233333164623135303132346164353165323338
-62306333363838393236346435303339393639623231353232643966613163313134336265633636
-64366539326639303362393933303465376532626663346431613563353032383037363532646637
-38376266323537343437643635343634636164393230313632663533666434306134366261383230
-66383135303565383763646331383338646339613539623835366438393635363832366134316531
-63636466653239383231616139663231623636376564376234383762383066613030313937616533
-33396135616236643034303236353230376434313365326633303961303232363766633161353434
-64333832623535343733353462313862343934633363616237353331386264386632643538333939
-34636437393934326131363236346636663665613537343134313636623439356665636663613938
-37346634356164343537396363663730306232623964636635333333653738383538343236383535
-66326462373964333234343435336630643930333433393034303439366631386666393330343238
-61336237373236313533333732343064336637326636663433396362346631343165656163313561
-30383331326231383539343639343130323731333432306630353835343338613861376631646264
-66383539616663376237663234313431313064663566313566366664386439316161353135306338
-35373962636537653535333935306564303235356232313463386237353737313736386463396164
-62666165336539316565623634326562356434313833643833353635303361353937666663356263
-35653665323736386263333437666432623332333936373938376637633463653930326466383139
-38336239646135316565356564373731386662383234386433633632393165343331353835623437
-30663138633430356334353832633434633939373337663437326263336537356633313763353337
-64373764666430643433
+32666334326535346466653862343263373365346261313136653366366664633436356536383437
+6666326436366433373163353262383031333264633530390a346539316165633939346663623264
+30653839376635626261323234326539313235623962306639396633343533363561376363363864
+3165323130633061350a346233396530303737316361653337643264383230613635383932636133
+61336561663264393831313364366665636166306437383835303332643934393062623163343861
+31643063373432643531663538666363613331626564376539336435343161333237333036343664
+31636635316631376563326637303337386337313330323838666562303463363361346531613363
+64383062396132373536663832356266383430316465643934636537396466366335636131623339
+34316166323730316463666465613734383339636264306431623535656362633739386362393134
+30343837363361333666616332646131663738663737663434303437336130313133306236313739
+36363731386163353136376230393262653334366239653032613034323333386433666262353364
+30326364343735643739626563393661616238313265633566356639623639383138303934653737
+64356139656163666263346539663733356239386333313763653434343163353436323937663034
+35653236616135383465656536323738346662396366653435383337383233643232656435356335
+63616138666662386337323865356439303961396138613965303465373766343939333933316664
+35356561353366663361306336303339633362623332386330383533336461306139613232306363
+62613731613835336338346236373636343463383433613938393865666165653935353830633966
+61393266623961343665343136323235393764313461306138343334366631313531396663666136
+34636436666238376138633461636438356461643461356139313437393736666338623038353933
+34633032653930623065623639646231346133343231383032663438326164313136613361356232
+38306565353737623532653231303963616336323563663137316139303136623234663432373661
+37333431663233656635373433323461663164303137633831306232626465303761633934336461
+65613534633665663565366634623362303964313862333936383032363137306336636563383435
+35366666313065393463656430303436346334313930616166383861613830373962353536303866
+39636165393861363161633734343838346338303036306639316231313738383831626639303833
+62316336623135336462633439323533383931666563643839383236636134363466383561633363
+64366466336362376234336463383639323165306631633437333639333263373837666565306163
+34666162336166623831353436363831666561376562613732333938353638333134323133346435
+61663261373433333635643761373465653031303236613465643931356433656161373463316331
+34326163633037633737393338313931353130306134663839623332666365373663643330353864
+36366535313032333262373038333161336230303165646133303261333839376432393435613434
+39633830316236653962663464363637333433393665353066653230383339666130653135343332
+32623165386535323832626536316665396564623064633564663437613930313665616163326637
+61363538333761633264396430326539643833313830346565643937656137376464646664313636
+38626165306661353631393031383063646463363031633435373735653838636634316634323738
+38313832376265616631623761393666346133636135666664643661646130306237383134383934
+39643561363962323261343236393630656435653261633933656262623737613130353531386438
+39643335306436636238393734646238653465626232643161316539346435353130633138393863
+63343965346361353036646561636339303063393033633162666639303166303265333233386137
+38643835616131303263366539646130643630366139623032366532653031303336323463326263
+37313566633032313931313565616131376635643030373833323838393633653531653664333032
+33623566333731636462366133653837636238383832353362653230393164343135663062373438
+61613832336633343038373565316431643432626532636435653438383536336166383436393035
+66613037663739616337383136663034643433333933643862396330356533636439666536353338
+30313761633961383337353030333862396164376337393338303565663963633662633134653362
+30346563643537323431313137636364623231376364386364633739313438376663343339333330
+64623663303535393262376437626637316532643139353262393231376135663161663862646666
+65303463393966393633613964353132303863373565353131396435623634336364313864373465
+61376335313830383534643431383966616565346466633633613933643231303732373938313434
+38383731653938656331646239353330666266336566633232353835653239393232366634663165
+31313766643539373133396563633637336133633034633236393336663064626630376632336264
+63346136613165613063316264646561633430613630396263333562366430326432323264383233
+30616631616535666631303838326330363339343463303831376531396164333539386263343334
+64313363626363316266343735313263613038383464396564313336373831393936336237303361
+65346161323961336631366263353161616562633533323036383366613637333832323466326535
+36376661346133383461626461646237373531333532656239333566323433333139633966343465
+36653130646636336534353331623062313863666637306462343139333031313234633636613232
+64383663626566616465623938346531623362383839316361366566336166386461336634303339
+63363931303866376462303634633033386332373564393932616638323764383630643839393630
+38373533636561613131343138383765376332336434303930326630303735313462316666333861
+37346336613339643466636665326466386664376438646666363537306533303036613135303230
+65303733366163363964613135306538393234636262326639613439363063326237306337666564
+32303161373661633231333662303565383134333737343530616631363337396333346261646636
+38663439653435623236366437643334393432383636316664306530303133396561396435313936
+38316637393365343936636438666461326436353636323537343363623337656536613662393333
+32346638636532333266323036623932316265303134633265366634623336386138653165643662
+35396637636136386366663762346236343565623133623336363530653039363166383534313632
+39333162396534376335646664383362366663373531646132643462396131333932653464326532
+39326633616335633030373463303665383261323763303863383566386536646136613362323034
+35313064313533616262666439663266353136616466643562666131366636623166333030346333
+62353539396131666434356238646439326337343239366234383866613065636232613863323931
+38363339623634643433393135396264363835386135303061623964343832323936313536623634
+34396531643839623932663565656535396337353934373462336231356133303335643931323131
+33383030653362323830376435393134666339333036336333376539363033666666396534636131
+64353663663633313966336439643334616332616536336238613162363435323938313636303930
+66306135613735653437366164336237656331346365646265656362316639656162386239326632
+34613431653433306633666632363639306266303361383230316539306161396566346137363133
+30333131666530636264656263303639636161366134633964393834663436326539626264383034
+34636134663833333764613237663233346363643566353166316230396235373261393739316237
+38373466623635333432326531613637303839633033636166633364383964626134386261636536
+64616432343936363664383532323936386136393332653131343533653633323639356234353363
+62613631333864663435616633396537373035646238653565646535306163623262396136356663
+63383537653231633766613231316666303830336334323738626138366434323164643465663034
+64343965363738626638363733646136316338633161383330306363643461643664303734663863
+38656135616165366438623237333161363234356538356466646266316136313832666161626566
+65306234663338323263376437363961353033396532643132336261336662656136656530343532
+61353736626665386437376230633336656663356361626437393932633765336439613665666632
+34363763656164326565633433643836383234363962613535353230356461613133666232396336
+32316264353535353162373833666365633266376634366362646264616161363634363138303832
+30656465326562653430373962346438646138336364363833646165353636346161336336613565
+37303931393865356136333431656264666331313165373037656535333832643662343434396337
+37656263363161396438626437643839633834643265313963653337363463373530333930303163
+34353163333461353763643631303763623034333665373235656531393561363432633433393137
+63333438646565633533353466373930313831316132306662363831623732336435346334323331
+32373736396239613839396535316166386637613435643637323932303239643339613737653761
+63323062333430636161316437383035326531346466336263353761636661326363656631613063
+62623863616533333036643563333463366432386232386436356632323863663862613466613931
+35636538663263336333613664353934623338356262643832356163326161373166633833333838
+36373537363934393137663462633833373364356230356136646639633737636130396231633730
+61356536373064356665646335633435646164623164333964323135343536663162656564653961
+63626432653439373938333039386661363330373136633831656230336237376436313664653633
+61623830326330663235376534313135303566636131373835383534363730616433363536346666
+38326532376338393462363430643334343137613363626132356133363534643336663063623136
+66623261343932373662663134323538666334623764376131623530373065316338653535313165
+35313337303563373737313963303134303435643563616561323963613565646466353162353637
+62623961626630323432323133666636356531316362633331313630393761303030363736643163
+63333565616232366665356535373632626366366532633036323839393266613730373331343765
+39343132643734623162386162353438353165653537303033313439326132343036333739623832
+30316132396532626236633665333162653536633134313339306661613066343937323032376235
+64386161623639636361313865613536373338333931623438366338616338326536383232363564
+30356139373865313432366337646366613361393262333336643535656438383562343763636236
+38616263613834613030363033613839613334323964343436326136626565643838323064313037
+34663964646636363038393134343634663637663030636339613864316165666265383632373637
+32373063616130343762393832663030636334646263626236663730336135376461643134646337
+63346534353264663031336164623330633737333766633662613863343431333138363337326536
+38363937313632633537363261633133636266376534346430643230623663333464393262396564
+37323632653036653336623636646537326636383865383338366138653463336638363965356536
+30373664313863396237613363373236353966303862376561653562666231353930653233383733
+38303261343436393261653436626136353237666633613166623035636466346539363436616364
+36386438666165316137303932346435313736616434643337303038383166386135326132656265
+64633866363532333465623135323761393837623934316561633637613433316334376138343764
+64646262356631373865393363353465643733623535626434666639613965383239343736356231
+31306135666539646234333934326664353137643261613131646364363739333035396437316362
+62343531656234376438376266373932376233643630383836323032633537383961383537376631
+38303062623032636264666534313266363835636132646664366162303430393161313964616633
+32613063393965653438643766373434613237393736643635653465366533363535623932386433
+38653530316364323033383330373433663865393837653466376666633434343062376432633138
+64653365373062643431353533363764666339663334393432353466623062646632383163656365
+63633365326565353730333164303237383766386135376630613234333834626335643738376339
+31626161346562613439396263646361343464636136636663363766356263636630323366326131
+37356166343031376635346433363335373137303933343539303832613363656334663964333462
+31613531383166666261336439356434323263306364616536636333316131373930646262386366
+33333032643464313165336265326130333637323461643131393437326535316164356665313636
+61613237356331393463336431316539633133326334396134663035396531633538666331393132
+64393134643832376230656332633633396333313464333066656461366335343131613038656535
+32353037316132393936373834633535646631653464616331633830326131333361316464393762
+34306136306633323766353235353663666539663036383636393335336263353134303634626131
+31316334663733666530653330333263653135643539353439306165353339623032643633393632
+37653636663164646438626638643630393135623934646166666635316532366264623165616131
+64653736306431623639303564306431616263613966656563373861616339316466383761643439
+64323836303162613931313534393436353464333330353635663034396631636563616464653530
+34616164323362653231343433323338633930323035396630326461666461346338363039383737
+38666431373633376536323739316132316231353738306632336537633938653535316530346634
+65323937336635373736346438323133353333343235623463663264623838316239366165306535
+30323136626561636537393066383539303233366262396538383563353730356537623266623038
+33316135303833323663363733353634653763383166633663313465303838646266303264363339
+30363637396161656332323333653931336661393566663232326466646563356436306332333532
+35656331383661376538633664346537613161303135643937643161383034643532653961346533
+31396237653234363832383437613433326165636262333236393334613437386638636535616634
+34313330643139386232396332376135376332613333633938366533616435303232376366623237
+37323535383932623631363431383165323338306338633066373937323361383336616265303234
+32343132303366343866303837326566383865393764343130616366373536623766316238363430
+31633734633438313065336431316161636465333166653765653962326636663065336532393731
+33386139383466626565663666373664313030663132313233356136613435663265396332656633
+33636436646565643838643930343938326231623738393661393738343062303661353362646465
+31653335343638613238333636326337396438373263663666373263616662653634613036663364
+66323639336233623538663061366462663766383539333432336363613938346263636436336430
+36646639353163316661356636393034653934666563356335396631323166313532353361653630
+61353261656431613135313965373061313038653237626231366332306432633538656462323739
+38313831663861663738336564333731363937653663663031343261633538643535336264306230
+31656365326261633730613765393861376662656137623832363832633266346135643133633331
+39623836633337323934653464333731626361316638333565393435656132626631613739366535
+37653266333432386334333461653231346630646435343230636531326338396566396466376532
+32666632383430393064353861373033393538366266333339373436353932656333366264393936
+39373938346633616534643734313934643438333965323133313037373737303964643337653135
+31623038356263643438343635643530663362656630393164663634366461623464353532353239
+38623662666235383139663162343836316331396266313062316639373034376337393563643739
+65313564303962353061616265393135326266386266316538616430383966306163386533376137
+36393565336435636535376639623931616435353532356438666663383961356638353333366139
+63313962643638323862353738316233613862346532306631386536323261636235343565353563
+64613465343662663034643138623865363862316131363461343636333864656132626166393836
+32393737386166343630323861376562353337663030613263613664663938376337333533613432
+33343064636365643831373135623466613738303164326562386430386339653831313536366532
+32613430313461663662313966373238643934633938633764363365386139303337643533313961
+61646131653638616365346236616330363566636261636362363635323566353562326239393530
+33323934623530306436623963663735353337306162646638383934326530333163396165366139
+61653339363166376266643733653536636639653939346666663834376361373533363231636433
+35363665373031393231363161333036303239313032326134363432363034623762366637643830
+32353132353961343830303562613964376234303461623263343735336238353565353238363438
+61326164323265386431353130346364393466353334386336376139303036646634376331313561
+39386331366337646634366635623066313432623537323164646263323532303465343735326364
+33383162626139383232636362613831303832633562636136393138316334303832383666313963
+37643835663065613236303338353435663830343637326563343665616532633830623035336534
+39376536663937653939326261346263613265636363333531323931646633313837386238333435
+33646339626462386635316230636135626330643630633664636364313539626332643331356334
+32313736343031633631393662363638626137303331383263656335373966383335653133333932
+34393861303665613633366461666664376466383437303065653938323166633562643834356139
+32616564373731313134316661343839333830663338636132323265376133656337393632353931
+35633531383438656461613135323134623134393062356661326435323065616331656638306432
+62313130643763626336383762306535656238393961653537336465373064653432653665373362
+63666366343866613235376535313835623261313531653265373636383030613930353530346265
+64303531316239386630626565323966616535373936393766663063623464663864363338343466
+33666237613532326661626461343166323162366532386638333835663838343865346266646533
+66343533326562396331363137353631633965623561333963613565636264653031616339613232
+39666362643739663837326432623462646233306239326463616232383236346236383132396637
+34623332346533366362613465363263373339643033663264363332336463343337333466393866
+33333363313331376634363431366636376466336131363330323530306162666434393764393563
+35633530353839386362326262343635383434333338626563383463653263333430383465626535
+63363933343365326665343166633939313739316231383938376166646331623663343161623865
+36646630336337373865

ansible/inventory/group_vars/authentik.yml

@@ -0,0 +1,39 @@
+$ANSIBLE_VAULT;1.1;AES256
+32656264356562366161643464343065313163663461663038646531306532383532393238396566
+3831646133656131336266346531323035313234663336380a356536313965383066363135343739
+32663032373165643931663762333036393430333437366630326562653034636464336166313730
+6666366634346664330a643063343061623465393436656461303862336431393931643638333735
+36663561626134333639326635373735623739366366396564653763656661663866623639323233
+63326130303065663265356136343266613966333230653935343836343461623535363161346439
+64653130383539363933636437643035306536376439326539666662383064636337363265356135
+39623537303165356635373666643634383731333236643061343033376466643762373636333834
+65626435353563623938363566626330356662366666666263363038646334323731393631356531
+37346230303134326135633033393162333839346664363062633736363861343435623331343762
+36643266333965323861633836336234333839306330343961653636623439656432613662303865
+33313438316236653461346264373338393539326438666337346162326163313837626364623461
+37353135323866326536303138363165396339306237393064366335386339303238373563373064
+62646430313131373964346466373164306432363361626661653531356261313233616563353465
+33353661663930333037636637353234663231633339333436393362393363653066376337376364
+61663035396563643665353162356466623032643938353933393930656162386631376132383461
+65383265623136646261393863336635323634336136376534323231383930383632306161393136
+66366563633737613931633233666336363166313930613963396636633761633638323465306332
+65333764343762643133613036353937386338303336656530366433333031323336366165346435
+62626533363961636361306664616638396538646530633730373636643131303636333036326534
+65616361353435346161383366626334386536376630326334353837396161356637356431316364
+31643636326262383362336535346237666465363165643830613932336362623635366138303336
+66376362386531313063336266623033653736616430386635656134616461643238373831333062
+62343964343032366434316365626232656337613832313263353237353336643735353464656262
+66313765393332393663303562393438376139373565623734653432626134376630343863353061
+31383530643264643330313136356638646265663831363637363464636634356639313936336437
+33393961363930616435623864303732633564303937343638373761653761666339373433373432
+39653333396138383332353465396639346535666536373232653536393163353835613239386530
+62666536633636363035613063643965623735366333386335363066326238323835663535333162
+36613366326137303564653463393665666337363664363563643361373766663331333363326533
+36353265616261613232633833616362363433313562303536326537346662383239343965313964
+36353534666535313239316166343163343132323134313266653537646136383262626233376332
+31616166366234653632373732633061373362336137303831353763373166643031313466656434
+64623362663762316465633564633462363866373762383937386362623462333461666462613336
+64393633336137663631386562366339646464363433396366303466336561653938386639393563
+33613936356336346134323830366236326231653465373438623433613634656666386162336331
+32316637383637336266656637343639613233623739303438323033333935663261373230626661
+6635353534313662646665613438336264303835633564303166

ansible/inventory/group_vars/edge_host/vars.yml

@@ -1,7 +1,8 @@
 ---
 # Secret indirection — real values live in vault.yml (ansible-vault) next
-# to this file; see docs/edge-host.md for how to populate it from
+# to this file; see roles/edge-host/README.md for how to populate it from
 # the cluster secrets.
+edge_sml2mqtt_mqtt_password: "{{ vault_edge_sml2mqtt_mqtt_password }}"
 edge_open3e_mqtt_password: "{{ vault_edge_open3e_mqtt_password }}"
 edge_zwavejs_session_secret: "{{ vault_edge_zwavejs_session_secret }}"
 # Loki push credential (dedicated 'edge-1' user, Traefik basic-auth on the
@@ -35,18 +36,3 @@ keepalived_ntp_sysctl:
 
 # Realtek NIC on the Wyse 3040 has no hardware timestamping.
 chrony_hwtimestamp: false
-
-# --- Local MQTT contract -------------------------------------------------
-# Shared between the broker and its clients, so it belongs to the inventory
-# rather than to either side's role. mqtt-bridge listens here; heating-can
-# publishes here; and the bridge authenticates to the cluster
-# broker AS the open3e user, so it needs that identity too.
-#
-# Keeping these in one role's defaults only worked while a single play loaded
-# every role at once. Splitting the plays by capability exposed it: the bridge
-# could not render without a heating-can default, and the CAN services could
-# not render without an mqtt-bridge default.
-edge_mqtt_host: 127.0.0.1
-edge_mqtt_port: 1883
-edge_open3e_mqtt_user: open3e
-edge_open3e_mqtt_topic: open3e

ansible/inventory/group_vars/n8n/vars.yml

@@ -0,0 +1,6 @@
+---
+# n8n group variables
+# Maps vault secrets to role defaults
+
+n8n_encryption_key: "{{ vault_n8n_encryption_key }}"
+n8n_postgres_password: "{{ vault_n8n_postgres_password }}"

ansible/inventory/group_vars/netbird_proxy/vault.yml

@@ -1,23 +0,0 @@
----
-# Bouncer key for crowdsec-edge-bouncer, scoped to the proxy hosts.
-#
-# A single encrypted VALUE rather than a whole-file vault like its siblings in
-# group_vars/all and group_vars/edge_host. Deliberate: adding this variable to
-# an existing whole-file vault means decrypting that file first, which exposes
-# every other secret in it for the sake of appending one line. encrypt_string
-# keeps the blast radius at this one value.
-#
-# Same value as OpenBao homelab/infrastructure/crowdsec-edge-bouncer-key, which
-# the crowdsec-edge-bouncer-key ExternalSecret feeds to BOUNCER_KEY_edge on the
-# LAPI. Both sides must move together — and unlike crowdsec-bouncer-key, this
-# one is NOT in the quarterly rotator, because nothing on these hosts re-reads
-# OpenBao on a schedule. Rotating it means updating OpenBao and re-running
-# plays/install-netbird-proxy.yml.
-vault_crowdsec_edge_bouncer_key: !vault |
-          $ANSIBLE_VAULT;1.1;AES256
-          32653764333536316163303838333839616562663862666637626165303837633866643033393239
-          3363376538656236643161386432326437633232643564630a633635306365323537346139626364
-          36613033343938363366306239363638383466396337623333623136313234656366333366623433
-          6437656632373364340a346566343866366433306533353431383064356362346536383136323663
-          39333864643534326433653233303137373162663066633132623931316135323733333863306330
-          6366623463666435636362376238313639666263323163303733

ansible/inventory/group_vars/ntp_cluster.yml

@@ -1,18 +0,0 @@
----
-# Shared configuration for every NTP server (see plays/install-ntp-ha.yml).
-
-# IPv6 service address -- what ntp.srv.renner.uno resolves to, and the path
-# clients actually use. The IPv4 VIP is legacy.
-#
-# Defined at group level rather than in the play because install-ntp-ha.yml runs
-# two plays over this group -- withdraw the static assignments fleet-wide, then
-# hand the address to keepalived -- and both need the same value. Duplicating a
-# service address across plays is exactly the sort of thing that drifts.
-ntp_vip6: "fd40::123/128"
-
-# Link-local address carried as the FIRST virtual_ipaddress of the IPv6 VRRP
-# instance. RFC 5798 gives an IPv6 virtual router a link-local primary, and
-# keepalived warns "the first IPv6 VIP address should be link local" without
-# one. fe80::123 mirrors the service address and cannot collide with a real
-# NIC's EUI-64 link-local.
-ntp_vip6_linklocal: "fe80::123/64"

ansible/inventory/group_vars/proxmox_all_lxc.yml

@@ -0,0 +1,2 @@
+---
+ansible_user: root

ansible/inventory/group_vars/proxmox_docker/vars.yml

@@ -0,0 +1,3 @@
+---
+# Docker hosts - tagged with 'docker' in Proxmox
+docker_monitoring_enabled: true

ansible/inventory/group_vars/pve_cluster.yml

@@ -0,0 +1,8 @@
+---
+# --- NTP (see plays/install-ntp-ha.yml) ---
+# The PVE hosts hold the 10.2.40.4 VIP via keepalived on the VLAN 40 bridge
+# sub-interface. This role disappears when these nodes are rebuilt as bare-metal
+# Talos, which cannot run chrony or keepalived — edge-1 is being brought into
+# ntp_cluster to take over.
+ntp_keepalived_enabled: true
+ntp_keepalived_interface: "vmbr1.40"

ansible/inventory/host_vars/grist.srv.renner.uno/vault.yml

@@ -0,0 +1,20 @@
+$ANSIBLE_VAULT;1.1;AES256
+66356134353363643130323632626530653863306635396432353539623731346138323635336632
+3536633563333231373333323564373763363936356535630a333135623336613832313833626531
+34646137633633353266653536363934303939626564353036343033643462373439643139353065
+6539383436343233300a396339356562363832373962306337663464363961346335656130653235
+38646563656538373530643531643964633934623465643065626266396130643966343166313937
+65663965613937323634323436386130653633656261653065306365316135343531653737663138
+65303339323137653362383766343361643837356531643339613431333238613564343563333761
+30616530613133633762306533363134383265313031343230313764633238626565623935313134
+38323435653264643534366366313862646161373434623135366164323236363733653533656233
+34336233396465626264666637363532646562306237363533616461613764306433366639623562
+30393234396238353038356636373632326130306431663439316363646530636563666364316139
+34326333363536343963393833616465303636323166643638623061353964303265393733366234
+33373435643237636536363364373830353833346235333534626561343465373236643165646664
+39396233363531393835656463666662616633643566383061323161663061323831356337333337
+65653336663735633536353263316634373639313238343365353833646666353131656530373361
+32643133353737366465633635663339666138323035633537653462303532366632626262373532
+32643132363062623033356134356135346361343234656636663936336338303230616665326630
+65353231356538633034626332633831396362373733646539313834386535666631613463656639
+373864663230353339666137653835656630

ansible/inventory/host_vars/loki.srv.renner.uno/vars.yml

@@ -0,0 +1,11 @@
+---
+# Loki server - central log aggregation host
+
+# Enable syslog receiver for remote devices (Synology NAS, etc.)
+# Synology DSM's "IETF" format actually sends BSD/RFC 3164 format: <pri>MMM DD HH:MM:SS host app: msg
+# Using raw format to accept messages and parse them with regex
+alloy_syslog_receiver_enabled: true
+alloy_syslog_receiver_port: 1514
+alloy_syslog_receiver_protocol: "udp"
+alloy_syslog_receiver_format: "raw"
+alloy_stability_level: "experimental"

ansible/inventory/host_vars/nas-1.renner.uno/vars.yml

@@ -0,0 +1,37 @@
+---
+# PBS Client configuration for nas-1
+pbs_backup_paths:
+  - /volume1/Documents
+  - /volume1/homes
+  - /volume1/Homeassistant Backup
+  - /volume1/Games
+  - /volume1/Media
+  - /volume1/Music
+  - /volume1/video
+  - /volume2/Backup
+  - /volume1/Movies
+
+  # Add more paths as needed:
+  # - /volume1/photos
+  # - /volume1/music
+
+# PBS Server connection
+pbs_server_host: "pbs2.renner.uno"
+pbs_server_port: 8007
+pbs_datastore: "pool"
+pbs_backup_id: "nas-1"
+
+# PBS Authentication - credentials in vault.yml
+pbs_username: "root@pam!nas-1"
+
+# Garage S3 (K8s backup target)
+garage_image: "dxflrs/garage:v2.3.0"
+garage_zone: "dsm"
+garage_capacity: "2T"
+
+# SSH connection - use password auth (key not supported by DSM)
+ansible_ssh_pass: "{{ nas_ssh_password }}"
+ansible_become_pass: "{{ nas_ssh_password }}"
+
+# Python 3.9+ from Synology package
+ansible_python_interpreter: /usr/local/bin/python3.9

ansible/inventory/host_vars/nas-1.renner.uno/vault.yml

@@ -0,0 +1,27 @@
+$ANSIBLE_VAULT;1.1;AES256
+34663462636437633437613863366266306334666635643839313737343739643864613136613938
+3138393966306561656366396465383539303137333036300a376237376636356230343834386362
+36636535396330333732383233323263613132333166343739396137636538653435336662353138
+6565643866623235610a316266383933303830323936303735623338623165386339383733373435
+62366461363833616438373237383132366438326364363462333265366533323861376630643365
+39343036343838333639363432643738666630363063323636646661313539643465366532363438
+33303837623234613434633563666562346536346435353461373536323737376233386163326537
+61383963343737656438356136613537623538333163343466343739346135333439616432663763
+36663962633037376561326434303035333938646436663832303432346365353439666435613337
+34363636326530333233313231343566306432376532396666643761633136316333376337666135
+65363338623035663438323338663863366362653131336434323865326530326534623631336263
+36633063383031386161363131333636393364623436613331353733313235383062656131313164
+63393665633361313961363637663237616364643236383835393736643937393564656238376362
+31393465383361326264376133393233616264663636383337363962356532613438663637616536
+33653137656335323763313461323633616635346232303062323062333766383931313965346361
+66656461313239633366636463633465636463663937393932636337643861383937663662306466
+32353166643832303234643264633062616330393963333034623432613730366339306637613239
+34626161663666316339323762373639353335653663353635366664643063393432623039306237
+38363161343930363763393364663364353735343236313665313461626165636135303839363164
+37336638396564326166376233353730346563656266323535666265323062646232323938353732
+32636235333839623166633761376334316434613235333730353439343262393537363461333238
+32653435623362666662336532643832356163326332636436346132643237643030343761663332
+36663530356539303636663438646430363335353135323932613863666264363533326137353164
+31663032346236616263343637396439663334396166656364623637666535386530383535663435
+66303037303265616634373763616437333835333136336563323339666563663931396133633131
+3963653266633535303630653335666166313336653831623362

ansible/inventory/host_vars/paperless.srv.renner.uno.yml

@@ -0,0 +1,8 @@
+---
+# Paperless server specific configuration
+
+# Extra scrape targets for Alloy
+alloy_extra_scrape_targets:
+  - name: "paperless"
+    address: "127.0.0.1:9199"
+    job: "paperless"

ansible/inventory/host_vars/wiki.srv.renner.uno/vars.yml

@@ -0,0 +1,7 @@
+---
+# Wiki server configuration
+
+# Home Assistant documentation generation (disabled - see HA Wiki Documentation project)
+homeassistant_docs_enabled: false
+homeassistant_url: "http://10.2.30.11:8123"
+homeassistant_token: "{{ vault_homeassistant_token }}"

ansible/inventory/pve_hosts.yml

@@ -1,16 +1,19 @@
-# Hosts are reached by name. An address in an inventory is allocator state
-# standing in for identity, and it pins access to the legacy family -- the names
-# resolve IPv6-first, which is the primary path on this network. DNS is the
-# authoritative source for where a host is; this file says which host, not where
-# it currently sits.
-#
-# Gone from this file, and why:
-#   pve_cluster (jupiter/neptune/saturn)  run Talos on bare metal; no SSH at all
-#   authentik_db / authentik_app          authentik runs in-cluster (CNPG)
-#   grafana                               runs in-cluster
-#   synology_nas (nas-1)                  powered off; Garage and the PBS client
-#                                         it hosted are gone with it
-# Their hosts no longer resolve, except the three that resolve to Talos nodes.
+[pve_cluster]
+jupiter ansible_host=10.2.40.201 ansible_user=root
+neptune ansible_host=10.2.40.202 ansible_user=root
+saturn ansible_host=10.2.40.203 ansible_user=root
+
+[authentik_db]
+authentik-db-1.srv.renner.uno ansible_user=root
+authentik-db-2.srv.renner.uno ansible_user=root
+
+[authentik:children]
+authentik_db
+authentik_app
+
+[authentik_app]
+authentik-app-1.srv.renner.uno ansible_user=root
+authentik-app-2.srv.renner.uno ansible_user=root
 
 [netbird_server]
 netbird.renner.uno ansible_user=root
@@ -19,47 +22,22 @@ netbird.renner.uno ansible_user=root
 proxy-1.renner.uno ansible_user=root
 proxy-2.renner.uno ansible_user=root
 
-# Every bare-metal edge thin client. Gets the host-generic roles only:
-# flash-wear handling, the log shipper, the MQTT bridge, the VLAN
-# sub-interface. Nothing here assumes any peripheral is attached.
+
+[synology_nas]
+nas-1.renner.uno ansible_user=ladmin ansible_python_interpreter=/usr/bin/python3
+
+[grafana]
+grafana.srv.renner.uno ansible_user=ladmin
+
 [edge_host]
 edge-1.iot.renner.uno ansible_user=root
 
-# Capability groups. Membership states which hardware a host physically has,
-# and each service playbook targets the group for its own concern. No flags and
-# no `when:` on the roles -- moving a stick to another box is a one-line change
-# here, which is what it should be.
-#
-# These are children of [edge_host], declared below: a peripheral is added to an
-# edge client, it does not replace the base. Making that a :children relation
-# rather than a comment means a host added only to a capability group still
-# resolves group_vars/edge_host -- where the local MQTT contract lives -- so it
-# cannot silently come up without edge_mqtt_host and friends.
-[zwave_gateway]
-edge-1.iot.renner.uno
-
-[heating_can]
-edge-1.iot.renner.uno
-
-[edge_host:children]
-zwave_gateway
-heating_can
-
-# NTP servers. edge-1 was added here so the role would survive the PVE hosts
-# being retired for bare-metal Talos, which cannot run chrony or keepalived
-# itself. That retirement has now happened, so edge-1 is the ONLY member.
-#
-# THIS IS NO LONGER HIGHLY AVAILABLE. The group was jupiter (MASTER, prio 100),
-# saturn (90), neptune (80) and edge-1 (BACKUP, 70); the first three are gone.
-# A single VRRP member takes the address regardless of its configured state, so
-# fd40::123 is still served — verified answering — but by one thin client with
-# no failover behind it. install-ntp-ha.yml still carries the four-host
-# ntp_servers priority map, which now describes hosts that do not exist.
-#
-# Restoring redundancy needs a second chrony host that is not a Talos node.
-# See plays/install-ntp-ha.yml for why keepalived is scoped more narrowly than
-# chrony within this group.
+# NTP servers. Includes edge-1 so the role survives the PVE hosts being
+# retired for bare-metal Talos, which cannot run chrony or keepalived itself.
+# See plays/install-ntp-ha.yml for why keepalived is scoped more narrowly
+# than chrony within this group.
 [ntp_cluster:children]
+pve_cluster
 edge_host
 
 # Off-site Debian VM (Synology VMM on nas-2, 10.3.40.0/24) running Claude Code.

ansible/plays/backup-authentik-db.yml

@@ -0,0 +1,48 @@
+---
+# Backup Authentik PostgreSQL database for K8s migration
+# Creates a pg_dump on the primary DB host
+#
+# Usage (via CI):
+#   ansible-playbook -i inventory/ plays/backup-authentik-db.yml
+#
+# Output: /tmp/authentik-backup.sql on the primary DB host
+
+- name: Backup Authentik PostgreSQL database
+  hosts: authentik-db-1.srv.renner.uno
+  become: true
+  tasks:
+    - name: Create database backup
+      become_user: postgres
+      command: >
+        pg_dump
+        --clean --if-exists
+        --no-owner --no-privileges
+        --format=custom
+        --file=/tmp/authentik-backup.dump
+        {{ authentik_db_name | default('authentik') }}
+      changed_when: true
+
+    - name: Make backup readable
+      file:
+        path: /tmp/authentik-backup.dump
+        mode: "0644"
+
+    - name: Get backup size
+      stat:
+        path: /tmp/authentik-backup.dump
+      register: backup_stat
+
+    - name: Report backup size
+      debug:
+        msg: "Backup created: {{ (backup_stat.stat.size / 1024 / 1024) | round(1) }} MB"
+
+    - name: Fetch backup to controller
+      fetch:
+        src: /tmp/authentik-backup.dump
+        dest: /tmp/authentik-backup.dump
+        flat: true
+
+    - name: Clean up remote backup
+      file:
+        path: /tmp/authentik-backup.dump
+        state: absent

ansible/plays/backup-paperless-db.yml

@@ -0,0 +1,59 @@
+---
+# Dump Paperless-ngx DB from the legacy VM to a local file on the
+# controller (CI runner), where a downstream pipeline step consumes it
+# to produce a Barman seed for the CNPG `bootstrap.recovery` path.
+# Data + media tarballs are handled separately by `backup-paperless.yml`.
+
+- name: Dump Paperless DB from legacy VM to CI runner
+  hosts: paperless.srv.renner.uno
+  become: true
+  vars:
+    paperless_db_name: paperless
+    paperless_db_user: paperless
+  tasks:
+    - name: Find running paperless DB container
+      ansible.builtin.shell: >
+        docker ps --filter "label=com.docker.compose.service=db"
+        --format '{{ "{{" }}.Names{{ "}}" }}' | head -1
+      register: paperless_db_container
+      changed_when: false
+
+    - name: Fail if DB container not found
+      ansible.builtin.fail:
+        msg: "Could not find paperless DB container (compose service=db)"
+      when: paperless_db_container.stdout | trim == ""
+
+    - name: Dump DB to /tmp on legacy host
+      ansible.builtin.shell: >
+        docker exec -i {{ paperless_db_container.stdout | trim }}
+        pg_dump --clean --if-exists --no-owner --no-privileges --format=custom
+        -U {{ paperless_db_user }} {{ paperless_db_name }}
+        > /tmp/paperless-backup.dump
+      args:
+        executable: /bin/bash
+      changed_when: true
+
+    - name: Make dump readable
+      ansible.builtin.file:
+        path: /tmp/paperless-backup.dump
+        mode: "0644"
+
+    - name: Report dump size
+      ansible.builtin.stat:
+        path: /tmp/paperless-backup.dump
+      register: dump_stat
+
+    - name: Show dump size
+      ansible.builtin.debug:
+        msg: "Dump created: {{ (dump_stat.stat.size / 1024 / 1024) | round(1) }} MB"
+
+    - name: Fetch dump to controller
+      ansible.builtin.fetch:
+        src: /tmp/paperless-backup.dump
+        dest: /tmp/paperless-backup.dump
+        flat: true
+
+    - name: Remove remote dump
+      ansible.builtin.file:
+        path: /tmp/paperless-backup.dump
+        state: absent

ansible/plays/check-dist-upgrade.yml

@@ -0,0 +1,95 @@
+---
+- name: Check for Distribution Upgrades
+  hosts: proxmox_all_qemu:&status_running:!proxmox_noansible
+  become: true
+  gather_facts: true
+  ignore_unreachable: true
+  
+  tasks:
+    - name: Check Ubuntu Release Availability
+      ansible.builtin.command: do-release-upgrade -c
+      register: ubuntu_check
+      changed_when: false
+      failed_when: false
+      when: 
+        - ansible_distribution is defined
+        - ansible_distribution == 'Ubuntu'
+
+    - name: Determine Upgrade Status
+      ansible.builtin.set_fact:
+        upgrade_needed: >-
+          {%- if ansible_distribution is not defined -%}
+            unknown
+          {%- elif ansible_distribution == 'Ubuntu' -%}
+            {%- if ubuntu_check.stdout is defined and 'New release' in ubuntu_check.stdout -%}
+              true
+            {%- else -%}
+              false
+            {%- endif -%}
+          {%- elif ansible_distribution == 'Debian' -%}
+            {%- if ansible_distribution_major_version | int < 12 -%}
+              true
+            {%- else -%}
+              false
+            {%- endif -%}
+          {%- else -%}
+            false
+          {%- endif -%}
+        current_version_str: "{{ ansible_distribution }} {{ ansible_distribution_version }}"
+        target_version_str: >-
+          {%- if ansible_distribution is not defined -%}
+            Unknown
+          {%- elif ansible_distribution == 'Ubuntu' -%}
+            {%- if ubuntu_check.stdout is defined and 'New release' in ubuntu_check.stdout -%}
+              {{ (ubuntu_check.stdout | regex_findall("New release '([^']+)'") | first) | default('Unknown') }}
+            {%- else -%}
+              -
+            {%- endif -%}
+          {%- elif ansible_distribution == 'Debian' -%}
+            {%- if ansible_distribution_major_version | int < 12 -%}
+              Debian 12 (Bookworm)
+            {%- else -%}
+              -
+            {%- endif -%}
+          {%- else -%}
+            -
+          {%- endif -%}
+      when: ansible_distribution is defined
+
+    - name: Group hosts by status
+      ansible.builtin.group_by:
+        key: "upgrade_{{ upgrade_needed | trim }}"
+      when: upgrade_needed is defined
+
+- name: Generate Report
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Display Upgrade Candidates
+      ansible.builtin.debug:
+        msg: "{{ report.splitlines() }}"
+      vars:
+        report: |
+          
+          DISTRIBUTION UPGRADE REPORT
+          ===========================
+
+          REQUIRES UPGRADE
+          ================
+          {{ '%-30s' | format('HOST') }}   {{ '%-20s' | format('CURRENT') }}   {{ 'TARGET' }}
+          {{ '------------------------------' }}   {{ '--------------------' }}   {{ '--------------------' }}
+          {% for host in groups['upgrade_true'] | default([]) %}
+          {{ '%-30s' | format(host) }}   {{ '%-20s' | format(hostvars[host]['current_version_str']) }}   {{ hostvars[host]['target_version_str'] }}
+          {% endfor %}
+          {% if groups['upgrade_true'] is not defined or groups['upgrade_true'] | length == 0 %}
+          (No hosts require upgrade)
+          {% endif %}
+
+          UP TO DATE
+          ==========
+          {{ '%-30s' | format('HOST') }}   {{ 'CURRENT' }}
+          {{ '------------------------------' }}   {{ '--------------------' }}
+          {% for host in groups['upgrade_false'] | default([]) %}
+          {{ '%-30s' | format(host) }}   {{ hostvars[host]['current_version_str'] }}
+          {% endfor %}
+

ansible/plays/check-template.yml

@@ -0,0 +1,8 @@
+- name: Ensure debian-12 template exists on saturn
+  hosts: saturn
+  become: true
+  tasks:
+    - name: Download debian-12 template
+      command: pveam download local debian-12-standard_12.12-1_amd64.tar.zst
+      args:
+        creates: /var/lib/vz/template/cache/debian-12-standard_12.12-1_amd64.tar.zst

ansible/plays/common.yml

@@ -1,16 +1,22 @@
 ---
 - name: Install Custom CA Certificates
-  hosts: all
+  hosts: all:!proxmox_noansible
   become: true
   roles:
     - certificates
 
 - name: Base System Hardening & Common Configuration
-  hosts: all
+  hosts: all:!pve:!pbs:!proxmox_noansible:!proxmox_nodes:!pve_cluster
   become: true
   ignore_unreachable: true
   gather_facts: true
 
+  vars:
+    # Monitoring agent endpoints
+    loki_endpoint: "http://loki.srv.renner.uno:3100"
+    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
+
   roles:
     - common
     - qemu-guest-agent
+    - monitoring-agent

ansible/plays/configure-audiobookshelf.yml

@@ -0,0 +1,44 @@
+---
+# Configure Audiobookshelf permissions
+# Fixes upload issues by ensuring audiobookshelf user can write to storage directories
+
+- name: Configure Audiobookshelf
+  hosts: audiobookshelf.srv.renner.uno
+  become: true
+
+  tasks:
+    - name: Ensure audiobookshelf user exists
+      ansible.builtin.user:
+        name: audiobookshelf
+        system: true
+        create_home: false
+        state: present
+
+    - name: Set ownership on audiobooks directory
+      ansible.builtin.file:
+        path: /audiobooks
+        owner: audiobookshelf
+        group: audiobookshelf
+        mode: "0755"
+        state: directory
+        recurse: false
+
+    - name: Set ownership on audiobooks content recursively
+      ansible.builtin.command:
+        cmd: chown -R audiobookshelf:audiobookshelf /audiobooks
+      changed_when: true
+
+    - name: Set ownership on metadata directory
+      ansible.builtin.file:
+        path: /metadata
+        owner: audiobookshelf
+        group: audiobookshelf
+        mode: "0755"
+        state: directory
+        recurse: true
+
+    - name: Restart audiobookshelf service
+      ansible.builtin.systemd:
+        name: audiobookshelf
+        state: restarted
+        enabled: true

ansible/plays/configure-grafana-authentik.yml

@@ -0,0 +1,40 @@
+---
+# Configure Grafana to use Authentik for authentication
+#
+# Prerequisites:
+# 1. Create OAuth2/OpenID application in Authentik:
+#    - Name: Grafana
+#    - Slug: grafana
+#    - Client type: Confidential
+#    - Redirect URIs: https://dash.renner.uno/login/generic_oauth
+#    - Signing Key: authentik Self-signed Certificate
+#
+# 2. Add to vault (ansible/inventory/group_vars/all/vault.yml):
+#    vault_grafana_authentik_client_id: "<client-id-from-authentik>"
+#    vault_grafana_authentik_client_secret: "<client-secret-from-authentik>"
+#
+# 3. (Optional) Create groups in Authentik:
+#    - "Grafana Admins" - gets Admin role
+#    - "Grafana Editors" - gets Editor role
+#    - Others get Viewer role
+
+- name: Configure Grafana Authentik Authentication
+  hosts: grafana.srv.renner.uno
+  become: yes
+  roles:
+    - grafana-authentik
+
+  post_tasks:
+    - name: Display configuration status
+      debug:
+        msg: |
+          Grafana Authentik OAuth configured!
+
+          Users can now login at: {{ grafana_root_url }}
+
+          Role mapping:
+          - Members of "Grafana Admins" -> Admin
+          - Members of "Grafana Editors" -> Editor
+          - All others -> Viewer
+
+          If auto_login is enabled, users will be redirected to Authentik automatically.

ansible/plays/configure-pve-k8s-zfs-tuning.yml

@@ -0,0 +1,68 @@
+---
+# Set sync=disabled on the zvols backing the K8s control-plane VM root disks
+# (k8s-1/2/3, VMIDs 150/151/152) to eliminate ZFS fsync amplification on the
+# shared local-zfs pool. Trades a small durability window for major fsync
+# latency reduction on etcd's WAL writes:
+#
+#   - etcd's WAL fsync returns immediately; ZFS still flushes via txgs
+#     (default ~5s)
+#   - Worst case on hard power loss: up to ~5s of writes lost per CP member
+#   - With 3-member quorum and Velero backups, this is acceptable for homelab
+#   - The whole zvol is sync=disabled (Talos OS, containerd, kubelet
+#     ephemeral, etcd WAL all share it), but etcd is the dominant fsync user;
+#     other writers tolerate small loss windows
+#
+# Each PVE host owns exactly one CP zvol (jupiter→150, saturn→151, neptune→152).
+# Auto-discovery handles whatever the host has — no need to update if a CP
+# VM ever moves.
+#
+# Idempotent: skips zvols that already have sync=disabled.
+#
+# Run manually against freshly-created CP zvols so they get the property
+# before Talos starts writing (previously wired into the now-removed
+# rebuild-k8s pipeline between `recreate-vms` and `talos-bootstrap`).
+
+- name: Set sync=disabled on K8s CP VM zvols
+  hosts: pve_cluster
+  become: yes
+  gather_facts: no
+
+  tasks:
+    - name: Discover CP-VM zvols on this host
+      shell: |
+        set -o pipefail
+        zfs list -H -t volume -o name | grep -E "/vm-(150|151|152)-disk-[0-9]+$" || true
+      args:
+        executable: /bin/bash
+      register: cp_zvols
+      changed_when: false
+
+    - name: Read current sync property
+      shell: "zfs get -H -o value sync {{ item }}"
+      register: current_sync
+      changed_when: false
+      loop: "{{ cp_zvols.stdout_lines }}"
+      when: cp_zvols.stdout_lines | length > 0
+
+    - name: Set sync=disabled where not already
+      shell: "zfs set sync=disabled {{ item.item }}"
+      when:
+        - cp_zvols.stdout_lines | length > 0
+        - item.stdout | trim != "disabled"
+      loop: "{{ current_sync.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item }}"
+
+    - name: Verify sync=disabled
+      shell: "zfs get -H -o value sync {{ item }}"
+      register: verify_sync
+      changed_when: false
+      failed_when: verify_sync.stdout | trim != "disabled"
+      loop: "{{ cp_zvols.stdout_lines }}"
+      when: cp_zvols.stdout_lines | length > 0
+
+    - name: Summary
+      debug:
+        msg: >-
+          {{ inventory_hostname }}: tuned {{ cp_zvols.stdout_lines | length }}
+          zvol(s) [{{ cp_zvols.stdout_lines | join(', ') }}]

ansible/plays/configure-pve-node-network.yml

@@ -0,0 +1,53 @@
+---
+# Configure Proxmox VE Node Host Networking
+#
+# Fixes two defects found by live inspection on 2026-07-19:
+#
+#   1. All three nodes had no IPv6 address on any external interface. FRR
+#      enables ipv6 forwarding for the OSPFv3 corosync mesh, and with the
+#      kernel default accept_ra=1 the nodes silently discarded the Router
+#      Advertisements the gateway sends on VLAN 40. Fixed with accept_ra=2.
+#
+#   2. saturn's vmbr0 sits at MTU 1500 while its own uplink and both peer
+#      nodes are at 9000, so large frames between saturn's guests and the
+#      other nodes' guests blackhole instead of fragmenting.
+#
+# The sysctl fix is non-disruptive and applies on every run. The MTU fix is
+# staged as a pending PVE network change and only applied when explicitly
+# requested, because applying it reloads networking on the node and briefly
+# interrupts the Talos k8s VM and its Ceph OSD.
+#
+# Safe (converges IPv6, stages the MTU change without applying it):
+#   ansible-playbook -i inventory/ plays/configure-pve-node-network.yml
+#
+# Applying the staged MTU change, one node at a time:
+#   ansible-playbook -i inventory/ plays/configure-pve-node-network.yml \
+#     --limit saturn -e pve_node_network_apply_pending=true
+#
+# See wiki: pve-ipv6-accept-ra-forwarding-defect,
+#           pve-saturn-vmbr0-mtu-asymmetry
+
+- name: Configure PVE node host networking
+  hosts: pve_cluster
+  become: yes
+  serial: 1
+
+  roles:
+    - role: pve-node-network
+
+  post_tasks:
+    - name: Summarise node IPv6 state
+      ansible.builtin.command: ip -6 addr show dev vmbr1.40 scope global
+      register: pve_node_network_v6_summary
+      changed_when: false
+      failed_when: false
+
+    - name: Display node IPv6 state
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }}:
+          {{ pve_node_network_v6_summary.stdout_lines
+             | select('search', 'inet6')
+             | map('trim')
+             | list
+             | default(['no global IPv6'], true) }}

ansible/plays/edge-mqtt-bridge.yml

@@ -1,33 +0,0 @@
----
-# Local MQTT bridge and its credential receiver only (REN-1307). Run by
-# .woodpecker/deploy-edge-mqtt-bridge.yml on merge, so a change to this role
-# reaches edge-1 without a play run by hand. The full base play
-# (install-edge-host.yml) carries the same role.
-- name: Converge the edge MQTT bridge
-  hosts: edge_host
-  become: true
-
-  # The SML power-meter reader (sml2mqtt) is retired for good: owner decision
-  # 2026-09-19, grid data comes from the E380 on the CAN bus. Its role and
-  # play are gone; this removes the leftover compose project from edge-1 on
-  # the next pipeline run. Idempotent: a no-op once the directory is gone.
-  pre_tasks:
-    - name: Check for the retired sml2mqtt compose project
-      ansible.builtin.stat:
-        path: /opt/edge-host/sml2mqtt/docker-compose.yml
-      register: edge_sml2mqtt_compose
-
-    - name: Stop the retired sml2mqtt container
-      ansible.builtin.command:
-        cmd: docker compose down --remove-orphans
-        chdir: /opt/edge-host/sml2mqtt
-      when: edge_sml2mqtt_compose.stat.exists
-      changed_when: true
-
-    - name: Remove the retired sml2mqtt directory
-      ansible.builtin.file:
-        path: /opt/edge-host/sml2mqtt
-        state: absent
-
-  roles:
-    - mqtt-bridge

ansible/plays/install-authentik-ha.yml

@@ -0,0 +1,20 @@
+---
+- name: Setup PostgreSQL HA Cluster
+  hosts: authentik_db
+  become: true
+  vars:
+    # Set this in inventory or pass as extra var.
+    # For initial setup, we define authentik-db-1 as primary.
+    postgres_primary_host: "authentik-db-1.srv.renner.uno"
+    
+  roles:
+    - common
+    - postgres-simple-ha
+
+- name: Setup Authentik Stateless App
+  hosts: authentik_app
+  become: true
+  roles:
+    - common
+    - docker-install
+    - authentik-stateless

ansible/plays/install-blackbox-exporter.yml

@@ -0,0 +1,19 @@
+---
+# Deploy Blackbox Exporter to Prometheus server for endpoint probing
+# Usage: ansible-playbook -i inventory/ plays/install-blackbox-exporter.yml
+
+- name: Install Blackbox Exporter
+  hosts: prometheus.srv.renner.uno
+  become: yes
+  roles:
+    - blackbox-exporter
+
+  post_tasks:
+    - name: Verify blackbox_exporter is running
+      uri:
+        url: "http://127.0.0.1:9115/metrics"
+        status_code: 200
+      register: exporter_health
+      retries: 3
+      delay: 5
+      until: exporter_health.status == 200

ansible/plays/install-claude-workstation.yml

@@ -10,7 +10,7 @@
 - name: Deploy the Claude Code workstation
   hosts: claude_workstation
   become: true
-  # qemu-guest-agent is applied
+  # qemu-guest-agent + monitoring-agent (Alloy -> Loki/Prometheus) are applied
   # to every inventory host by the umbrella plays/common.yml, same as for the
   # netbird-proxy host — run that against this host too (see RUNBOOK Step 5).
   roles:

ansible/plays/install-edge-host.yml

@@ -1,41 +1,13 @@
 ---
-# Bare-metal Arch edge hosts (Dell Wyse 3040 and similar) — the home for
-# hardware-bound services, because Talos supports neither SocketCAN nor USB
-# passthrough.
-#
-# This play is the BASE only: what every edge thin client gets regardless of
-# what is plugged into it. The peripherals live in their own playbooks against
-# their own inventory groups:
-#
-#   plays/install-zwave-gateway.yml   [zwave_gateway]   ZWA-2 radio
-#   plays/install-heating-can.yml     [heating_can]     Innomaker USB2CAN
-#
-# Splitting the roles without splitting the plays would have kept the coupling:
-# one play applying every role to one group is the monolith again, a layer up.
-# A host with no peripherals is simply absent from those groups -- no flags, no
-# `when:` on the roles.
-#
-# Self-contained: `certificates` and `common` are the parts of plays/common.yml
-# that apply to an edge host, carried here so bootstrapping one is a single
-# unrestricted run over the group. Not carried over: qemu-guest-agent, since
-# these are bare metal rather than VMs. Log shipping is edge-log-shipper's job
-# and lives below — it pushes to the authenticated cluster ingress with a
-# per-host credential, which is the only Loki these hosts can reach.
-#
-# Order is dependency order: edge-flash-wear before docker, so the journal is
-# volatile before containers start logging into it.
-- name: Install Arch edge host (base)
+# Bare-metal Arch Linux edge host (Dell Wyse 3040, internal eMMC) — replaces
+# the k8s-edge Talos VM
+# (opentofu vm_153) as the home for hardware-bound services:
+#   - zwave-js-ui  (ZWA-2 Z-Wave USB radio, docker compose)
+#   - sml2mqtt     (FTDI power-meter reader, docker compose)
+#   - open3e       (Innomaker USB2CAN -> Vitocal/VX3, native venv + systemd)
+# Base install procedure and cutover checklist: ansible/roles/edge-host/README.md
+- name: Install Arch edge host
   hosts: edge_host
-  # Connection is already root (ansible_user=root), so this is a root-to-root
-  # no-op today. Kept explicit, matching plays/common.yml, so the roles below
-  # do not silently depend on the connection user staying privileged.
-  become: true
-
   roles:
-    - certificates
-    - common
-    - edge-flash-wear
     - docker-install
-    - edge-log-shipper
-    - mqtt-bridge
-    - edge-vlan
+    - edge-host

ansible/plays/install-garage-synology.yml

@@ -0,0 +1,313 @@
+---
+# Synology-specific playbook using raw module (Python 3.8 workaround)
+# Deploys Garage (https://garagehq.deuxfleurs.fr) as a single-node S3 target
+# for K8s backups (Velero + CNPG Barman Cloud Plugin).
+#
+# Required secrets in host_vars/nas-1.renner.uno/vault.yml:
+#   garage_rpc_secret:     "<openssl rand -hex 32>"
+#   garage_admin_token:    "<openssl rand -base64 32>"
+#   garage_metrics_token:  "<openssl rand -base64 32>"
+#
+# Post-install bootstrap (run once, record the returned secret keys):
+#   docker exec garaged /garage status
+#   docker exec garaged /garage bucket create k8s-velero
+#   docker exec garaged /garage bucket create k8s-cnpg
+#   docker exec garaged /garage bucket create k8s-secrets
+#   docker exec garaged /garage bucket create k8s-migration
+#   docker exec garaged /garage key create velero-key
+#   docker exec garaged /garage key create cnpg-key
+#   docker exec garaged /garage key create secrets-key
+#   docker exec garaged /garage key create migration-key
+#   docker exec garaged /garage bucket allow --read --write --owner k8s-velero    --key velero-key
+#   docker exec garaged /garage bucket allow --read --write --owner k8s-cnpg      --key cnpg-key
+#   docker exec garaged /garage bucket allow --read --write --owner k8s-secrets   --key secrets-key
+#   docker exec garaged /garage bucket allow --read --write --owner k8s-migration --key migration-key
+- name: Install Garage S3 on Synology NAS
+  hosts: synology_nas
+  gather_facts: false
+
+  vars:
+    garage_data_dir: "/volume1/docker/garage"
+
+  tasks:
+    - name: Create Garage directory tree
+      ansible.builtin.raw: |
+        mkdir -p {{ garage_data_dir }}/config \
+                 {{ garage_data_dir }}/meta \
+                 {{ garage_data_dir }}/data
+      changed_when: true
+
+    # Secret writes use the same compare-then-write pattern as garage.toml so
+    # rotation is idempotent AND a real change can drive force-recreate below.
+    # Garage reads these files once at startup; without container restart, a
+    # rotated token leaves the running process holding the previous value.
+    - name: Write rpc_secret
+      ansible.builtin.raw: |
+        cat > {{ garage_data_dir }}/config/rpc_secret.new << 'SECRET_EOF'
+        {{ garage_rpc_secret }}
+        SECRET_EOF
+        chmod 600 {{ garage_data_dir }}/config/rpc_secret.new
+        if [ ! -f {{ garage_data_dir }}/config/rpc_secret ] \
+        || ! cmp -s {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret; then
+          mv {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret
+          echo RPC_SECRET_CHANGED
+        else
+          rm {{ garage_data_dir }}/config/rpc_secret.new
+        fi
+      register: rpc_secret_write
+      changed_when: "'RPC_SECRET_CHANGED' in rpc_secret_write.stdout"
+      no_log: true
+
+    - name: Write admin_token
+      ansible.builtin.raw: |
+        cat > {{ garage_data_dir }}/config/admin_token.new << 'SECRET_EOF'
+        {{ garage_admin_token }}
+        SECRET_EOF
+        chmod 600 {{ garage_data_dir }}/config/admin_token.new
+        if [ ! -f {{ garage_data_dir }}/config/admin_token ] \
+        || ! cmp -s {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token; then
+          mv {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token
+          echo ADMIN_TOKEN_CHANGED
+        else
+          rm {{ garage_data_dir }}/config/admin_token.new
+        fi
+      register: admin_token_write
+      changed_when: "'ADMIN_TOKEN_CHANGED' in admin_token_write.stdout"
+      no_log: true
+
+    - name: Write metrics_token
+      ansible.builtin.raw: |
+        cat > {{ garage_data_dir }}/config/metrics_token.new << 'SECRET_EOF'
+        {{ garage_metrics_token }}
+        SECRET_EOF
+        chmod 600 {{ garage_data_dir }}/config/metrics_token.new
+        if [ ! -f {{ garage_data_dir }}/config/metrics_token ] \
+        || ! cmp -s {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token; then
+          mv {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token
+          echo METRICS_TOKEN_CHANGED
+        else
+          rm {{ garage_data_dir }}/config/metrics_token.new
+        fi
+      register: metrics_token_write
+      changed_when: "'METRICS_TOKEN_CHANGED' in metrics_token_write.stdout"
+      no_log: true
+
+    - name: Write garage.toml
+      ansible.builtin.raw: |
+        cat > {{ garage_data_dir }}/config/garage.toml.new << 'TOML_EOF'
+        metadata_dir = "/var/lib/garage/meta"
+        data_dir     = "/var/lib/garage/data"
+
+        # SQLite chosen over LMDB: single-node deployment has no peers to
+        # rebuild metadata from on unclean-shutdown corruption. SQLite is
+        # slower but portable and has no LMDB corruption window.
+        db_engine = "sqlite"
+
+        # 6-hour auto-snapshot of the metadata DB as a backstop against
+        # corruption; last two snapshots retained.
+        metadata_auto_snapshot_interval = "6h"
+
+        # Single-zone, single-node: only coherent replication factor.
+        replication_factor = 1
+
+        rpc_bind_addr    = "[::]:3901"
+        rpc_secret_file  = "/etc/garage/rpc_secret"
+
+        compression_level = 2
+
+        # 64 MB blocks (default 1 MB; previously 20 MB). On the 5 × 7 TB
+        # Btrfs RAID 5 volume the per-block fixed cost (SQLite lookup,
+        # dir traversal, hash verify, zstd decompress, nginx buffering)
+        # is the dominant overhead per object; larger blocks amortize
+        # it. block_size is a maximum, so 16 MiB CNPG WAL segments
+        # still upload as one block each. Only applies to NEW uploads;
+        # existing objects keep their original chunking until re-uploaded.
+        block_size = "64M"
+
+        [s3_api]
+        api_bind_addr = "[::]:3900"
+        s3_region     = "garage"
+        root_domain   = ".s3.{{ inventory_hostname }}"
+
+        [s3_web]
+        bind_addr   = "[::]:3902"
+        root_domain = ".web.{{ inventory_hostname }}"
+
+        [admin]
+        api_bind_addr      = "[::]:3903"
+        admin_token_file   = "/etc/garage/admin_token"
+        metrics_token_file = "/etc/garage/metrics_token"
+        TOML_EOF
+        if [ ! -f {{ garage_data_dir }}/config/garage.toml ] \
+        || ! cmp -s {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml; then
+          mv {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml
+          echo GARAGE_TOML_CHANGED
+        else
+          rm {{ garage_data_dir }}/config/garage.toml.new
+        fi
+      register: garage_toml_write
+      changed_when: "'GARAGE_TOML_CHANGED' in garage_toml_write.stdout"
+
+    - name: Write docker-compose.yml
+      ansible.builtin.raw: |
+        cat > {{ garage_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
+        services:
+          garage:
+            image: {{ garage_image }}
+            container_name: garaged
+            restart: unless-stopped
+            # HTTP services (3900/3902/3903) bind to loopback; DSM
+            # reverse proxy fronts and TLS-terminates them. RPC 3901
+            # is exposed directly — the Noise handshake is raw TCP,
+            # so DSM RP cannot front it. rpc_secret gates auth.
+            ports:
+              - "3901:3901"
+              - "127.0.0.1:3900:3900"
+              - "127.0.0.1:3902:3902"
+              - "127.0.0.1:3903:3903"
+            volumes:
+              - {{ garage_data_dir }}/config/garage.toml:/etc/garage.toml:ro
+              - {{ garage_data_dir }}/config/rpc_secret:/etc/garage/rpc_secret:ro
+              - {{ garage_data_dir }}/config/admin_token:/etc/garage/admin_token:ro
+              - {{ garage_data_dir }}/config/metrics_token:/etc/garage/metrics_token:ro
+              - {{ garage_data_dir }}/meta:/var/lib/garage/meta
+              - {{ garage_data_dir }}/data:/var/lib/garage/data
+        COMPOSE_EOF
+      changed_when: true
+
+    - name: Pull image and start Garage container
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        cd {{ garage_data_dir }} && docker compose pull && docker compose up -d
+      become: true
+      changed_when: true
+
+    # Garage does not reload config on SIGHUP and compose does not notice
+    # when a bind-mounted host file changes. Force-recreate the container
+    # when garage.toml OR any of the three secret files changed so the new
+    # config / rotated tokens are picked up.
+    - name: Force-recreate garaged when garage.toml or secrets changed
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        cd {{ garage_data_dir }} && docker compose up -d --force-recreate garage
+      become: true
+      when: garage_toml_write.changed
+            or rpc_secret_write.changed
+            or admin_token_write.changed
+            or metrics_token_write.changed
+      changed_when: true
+
+    - name: Wait for Garage to accept CLI calls
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        for i in $(seq 1 30); do
+          if docker exec garaged /garage status >/dev/null 2>&1; then
+            exit 0
+          fi
+          sleep 2
+        done
+        exit 1
+      become: true
+      changed_when: false
+
+    - name: Check whether cluster layout is already applied
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        docker exec garaged /garage status
+      become: true
+      register: garage_status
+      changed_when: false
+
+    - name: Apply single-node layout (idempotent — only on first run)
+      when: "'NO ROLE ASSIGNED' in garage_status.stdout"
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        NODE_ID=$(docker exec garaged /garage node id | head -n 1 | cut -d@ -f1)
+        docker exec garaged /garage layout assign "$NODE_ID" \
+          -z {{ garage_zone }} \
+          -c {{ garage_capacity }} \
+          -t {{ inventory_hostname_short }}
+        docker exec garaged /garage layout apply --version 1
+      become: true
+      changed_when: true
+
+    - name: Report Garage status
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        docker exec garaged /garage status
+      become: true
+      register: garage_status_final
+      changed_when: false
+
+    - name: Show Garage status
+      ansible.builtin.debug:
+        var: garage_status_final.stdout_lines
+
+    # Bootstrap buckets + keys. Idempotent: bucket_info / key_info return
+    # non-zero when missing, so we only create on first run. Access-key
+    # secrets are captured at create time and persisted to chmod-600 files
+    # under {{ garage_data_dir }}/config/keys/<key>.creds on nas-1. These
+    # files are the only post-install record of the secrets — Garage does
+    # not expose them again.
+    - name: Bootstrap Garage buckets, keys, and grants
+      ansible.builtin.shell: |
+        set -e
+        export PATH=$PATH:/usr/local/bin
+        KEYS_DIR={{ garage_data_dir }}/config/keys
+        mkdir -p "$KEYS_DIR"
+        chmod 700 "$KEYS_DIR"
+
+        bootstrap_pair() {
+          local BUCKET="$1"
+          local KEY="$2"
+          local CREDS_FILE="$KEYS_DIR/$KEY.creds"
+
+          if ! docker exec garaged /garage bucket info "$BUCKET" >/dev/null 2>&1; then
+            docker exec garaged /garage bucket create "$BUCKET"
+          fi
+
+          if ! docker exec garaged /garage key info "$KEY" >/dev/null 2>&1; then
+            CREATE_OUT=$(docker exec garaged /garage key create "$KEY")
+            KEY_ID=$(echo "$CREATE_OUT"   | awk '/^Key ID:/     {print $3}')
+            SECRET=$(echo "$CREATE_OUT"   | awk '/^Secret key:/ {print $3}')
+            umask 177
+            printf 'KEY_ID=%s\nSECRET_KEY=%s\n' "$KEY_ID" "$SECRET" > "$CREDS_FILE"
+          fi
+
+          docker exec garaged /garage bucket allow \
+            --read --write --owner \
+            "$BUCKET" --key "$KEY"
+        }
+
+        bootstrap_pair k8s-velero    velero-key
+        bootstrap_pair k8s-cnpg      cnpg-key
+        bootstrap_pair k8s-secrets   secrets-key
+        bootstrap_pair k8s-migration migration-key
+        bootstrap_pair tfstate       tfstate-key
+      become: true
+      changed_when: true
+      no_log: true
+
+    - name: Report bucket + key state (no secrets)
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        docker exec garaged /garage bucket list
+        echo "---"
+        docker exec garaged /garage key list
+        echo "---"
+        ls -l {{ garage_data_dir }}/config/keys/ 2>&1
+      become: true
+      register: garage_bootstrap_state
+      changed_when: false
+
+    - name: Show bucket + key state
+      ansible.builtin.debug:
+        var: garage_bootstrap_state.stdout_lines
+
+    - name: Post-install note
+      ansible.builtin.debug:
+        msg:
+          - "Buckets: k8s-velero, k8s-cnpg, k8s-secrets, k8s-migration, tfstate"
+          - "Keys:    velero-key, cnpg-key, secrets-key, migration-key, tfstate-key"
+          - "Secrets are written to {{ garage_data_dir }}/config/keys/<key>.creds on nas-1 (chmod 600, root-owned)."
+          - "SSH to nas-1 and `sudo cat` the file to retrieve KEY_ID + SECRET_KEY for Velero / CNPG / migration / tfstate config."

ansible/plays/install-gatus.yml

@@ -0,0 +1,8 @@
+---
+- name: Install and Configure Gatus
+  hosts: proxmox_gatus
+  become: true
+  roles:
+    - role: common
+    - role: docker-install
+    - role: gatus

ansible/plays/install-grist.yml

@@ -0,0 +1,8 @@
+---
+- name: Install and Configure Grist
+  hosts: grist
+  become: true
+  roles:
+    - role: common
+    - role: docker-install
+    - role: grist

ansible/plays/install-heating-can.yml

@@ -1,18 +0,0 @@
----
-# Heating system over the Viessmann E3 CAN bus, for hosts with the Innomaker
-# USB2CAN adapter attached.
-#
-# Targets [heating_can]: membership means the adapter is physically present.
-# Two roles because they are two concerns -- the interface is host network
-# state, the services consume it -- but one group, because both need the same
-# hardware.
-#
-# Requires the base play (plays/install-edge-host.yml) to have run: these
-# services publish to the local MQTT bridge it sets up.
-- name: Install heating CAN services
-  hosts: heating_can
-  become: true
-
-  roles:
-    - socketcan
-    - heating-can

ansible/plays/install-ical-proxy.yml

@@ -0,0 +1,8 @@
+---
+- name: Install and Configure ical-proxy
+  hosts: proxmox_ical_proxy
+  become: true
+  roles:
+    - role: common
+    - role: docker-install
+    - role: ical-proxy

ansible/plays/install-log-aggregation.yml

@@ -0,0 +1,86 @@
+---
+# Setup Log Aggregation Infrastructure
+# This playbook configures the complete logging stack:
+# 1. Loki server for log storage and querying
+# 2. Alloy agents on all VMs/CTs for log collection
+# 3. Syslog receiver for remote devices (Synology NAS, network devices)
+#
+# Usage: ansible-playbook -i inventory/ plays/setup-log-aggregation.yml
+#
+# To limit to specific hosts:
+#   --limit loki.srv.renner.uno          # Only configure Loki server
+#   --limit proxmox_all_running          # Only configure log agents
+
+- name: Setup Loki Log Aggregation Server
+  hosts: proxmox_logging
+  become: yes
+  gather_facts: yes
+
+  roles:
+    - common
+    - loki-server
+
+  post_tasks:
+    - name: Verify Loki is running
+      uri:
+        url: "http://127.0.0.1:3100/ready"
+        status_code: 200
+      register: loki_health
+      retries: 5
+      delay: 10
+      until: loki_health.status == 200
+
+    - name: Display Loki status
+      debug:
+        msg: "Loki is running and healthy at http://{{ inventory_hostname }}:3100"
+
+- name: Deploy Monitoring Agent to Loki Server (with syslog receiver)
+  hosts: proxmox_logging
+  become: yes
+  gather_facts: yes
+
+  vars:
+    loki_endpoint: "http://loki.srv.renner.uno:3100"
+    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
+    # Syslog receiver settings are defined in host_vars/loki.srv.renner.uno/vars.yml:
+    # - alloy_syslog_receiver_enabled: true
+    # - alloy_syslog_receiver_port: 1514
+    # - alloy_syslog_receiver_protocol: "udp"
+    # - alloy_syslog_receiver_format: "raw"
+    # - alloy_stability_level: "experimental"
+
+  roles:
+    - monitoring-agent
+
+  post_tasks:
+    - name: Display syslog receiver info
+      debug:
+        msg: |
+          Syslog receiver is listening on UDP port {{ alloy_syslog_receiver_port | default(1514) }}
+          Configure remote devices to send syslog to: {{ inventory_hostname }}:{{ alloy_syslog_receiver_port | default(1514) }}
+      when: alloy_syslog_receiver_enabled | default(false)
+
+- name: Deploy Monitoring Agent to All Running VMs/CTs
+  hosts: proxmox_all_running:!proxmox_logging:!proxmox_noansible
+  become: yes
+  gather_facts: yes
+  serial: 10  # Deploy to 10 hosts at a time to avoid overwhelming the network
+
+  vars:
+    loki_endpoint: "http://loki.srv.renner.uno:3100"
+    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
+
+  roles:
+    - monitoring-agent
+
+  post_tasks:
+    - name: Verify Alloy is running
+      systemd:
+        name: alloy
+        state: started
+      register: alloy_service
+
+    - name: Display agent status
+      debug:
+        msg: "Monitoring agent deployed successfully on {{ inventory_hostname }}"
+      when: alloy_service is defined

ansible/plays/install-loki.yml

@@ -0,0 +1,26 @@
+---
+# Install Loki Log Aggregation Server
+# Usage: ansible-playbook -i inventory/ plays/install-loki.yml
+
+- name: Install Loki Log Aggregation Server
+  hosts: loki.srv.renner.uno
+  become: yes
+  gather_facts: yes
+
+  roles:
+    - common
+    - loki-server
+
+  post_tasks:
+    - name: Verify Loki is running
+      uri:
+        url: "http://127.0.0.1:3100/ready"
+        status_code: 200
+      register: loki_health
+      retries: 5
+      delay: 10
+      until: loki_health.status == 200
+
+    - name: Display Loki status
+      debug:
+        msg: "Loki is running and healthy at http://{{ inventory_hostname }}:3100"

ansible/plays/install-mkdocs-wiki.yml

@@ -0,0 +1,19 @@
+---
+# Deploy MkDocs wiki from Obsidian vault
+#
+# Usage:
+#   ansible-playbook -i inventory plays/install-mkdocs-wiki.yml
+#
+- name: Deploy MkDocs Wiki
+  hosts: proxmox_wiki
+  become: true
+
+  pre_tasks:
+    - name: Ensure wiki group exists
+      ansible.builtin.group:
+        name: wiki
+        system: true
+        state: present
+
+  roles:
+    - role: mkdocs-wiki

ansible/plays/install-music-assistant.yml

@@ -0,0 +1,8 @@
+---
+- name: Install and Configure Music Assistant
+  hosts: proxmox_music_assistant
+  become: true
+  roles:
+    - role: common
+    - role: docker-install
+    - role: music-assistant

ansible/plays/install-n8n.yml

@@ -0,0 +1,8 @@
+---
+- name: Install and Configure n8n
+  hosts: n8n
+  become: true
+  roles:
+    - role: common
+    - role: docker-install
+    - role: n8n

ansible/plays/install-netbird-monitoring.yml

@@ -0,0 +1,38 @@
+---
+# Deploy monitoring agent to NetBird server (external datacenter)
+# Uses NetBird mesh IPs to reach Loki and Prometheus
+# Usage: ansible-playbook -i inventory/ plays/setup-netbird-monitoring.yml
+
+- name: Deploy Monitoring Agent to NetBird Server
+  hosts: netbird_server
+  become: yes
+  gather_facts: yes
+
+  vars:
+    # Use ARM64 binary for Hetzner ARM server
+    alloy_arch: "linux-arm64"
+
+    # Monitoring endpoints via NetBird mesh
+    loki_endpoint: "http://loki.nb.renner.uno:3100"
+    prometheus_remote_write_endpoint: "http://prometheus.nb.renner.uno:9090/api/v1/write"
+
+    # NetBird server labels
+    alloy_extra_labels:
+      environment: "production"
+      datacenter: "hetzner"
+      type: "netbird-server"
+      service: "netbird"
+
+  roles:
+    - monitoring-agent
+
+  post_tasks:
+    - name: Verify Alloy agent is running
+      systemd:
+        name: alloy
+        state: started
+      register: alloy_status
+
+    - name: Display agent status
+      debug:
+        msg: "Monitoring agent deployed to {{ inventory_hostname }} via NetBird mesh"

ansible/plays/install-netbird-proxy.yml

@@ -1,5 +1,5 @@
 ---
-# Deploy the NetBird reverse-proxy HA cluster members (the public edge).
+# Deploy a NetBird reverse-proxy HA cluster member (a second proxy host).
 # Usage: ansible-playbook -i inventory/ plays/install-netbird-proxy.yml
 #
 # Prerequisites:
@@ -7,11 +7,12 @@
 #   listed under [netbird_proxy] in inventory/pve_hosts.yml. Docker is
 #   installed by the (Arch-aware) docker-install role.
 # - DNS A/AAAA for the host's own name (e.g. proxy-1.renner.uno) -> its IP.
-# - Uses the existing vault_netbird_proxy_token (shared by every member) — no
-#   new token to generate; a shared token + same NB_PROXY_DOMAIN forms the
-#   cluster.
-# - AFTER deploy + verify: add this host's A/AAAA to proxy.renner.uno so
-#   traffic round-robins across the cluster.
+# - Uses the existing vault_netbird_proxy_token (shared with VPS-1) — no new
+#   token to generate; a shared token + same NB_PROXY_DOMAIN forms the cluster.
+# - AFTER deploy + verify: add this host's IP to the proxy domain so traffic
+#   round-robins across the cluster. Repoint proxy.renner.uno away from its
+#   CNAME to netbird.renner.uno (management host) to A/AAAA records listing
+#   BOTH proxy IPs (VPS-1 + this host).
 
 - name: Deploy NetBird reverse-proxy cluster member
   hosts: netbird_proxy
@@ -25,7 +26,3 @@
     - docker-install
     - netbird-proxy
     - netbird-proxy-watchdog
-    # Enforces the cluster's CrowdSec decisions here, where public traffic
-    # enters. Deployed in dry-run: it logs what it would block and creates no
-    # nftables rules until crowdsec_bouncer_mode is flipped per host.
-    - crowdsec-edge-bouncer

ansible/plays/install-netbird-server.yml

@@ -18,3 +18,4 @@
     - common
     - docker-install
     - netbird-server
+    - netbird-proxy-watchdog

ansible/plays/install-node-exporter-pve.yml

@@ -0,0 +1,41 @@
+---
+# Install Node Exporter and Chrony Exporter on Proxmox VE hosts
+# Usage: ansible-playbook -i inventory/ plays/install-node-exporter-pve.yml
+
+- name: Install Exporters on PVE Cluster
+  hosts: pve_cluster
+  become: yes
+  gather_facts: yes
+
+  roles:
+    - node-exporter
+    - chrony-exporter
+
+  post_tasks:
+    - name: Verify node-exporter is running
+      uri:
+        url: "http://127.0.0.1:9100/metrics"
+        status_code: 200
+      register: node_exporter_health
+      retries: 3
+      delay: 5
+      until: node_exporter_health.status == 200
+
+    - name: Verify chrony-exporter is running
+      uri:
+        url: "http://127.0.0.1:9123/metrics"
+        status_code: 200
+      register: chrony_exporter_health
+      retries: 3
+      delay: 5
+      until: chrony_exporter_health.status == 200
+
+    - name: Verify chrony metrics include stratum
+      uri:
+        url: "http://127.0.0.1:9123/metrics"
+        return_content: yes
+      register: chrony_metrics
+
+    - name: Check chrony exporter output
+      debug:
+        msg: "Chrony stratum metrics available: {{ 'chrony_tracking_stratum' in chrony_metrics.content }}"

ansible/plays/install-node-exporter.yml

@@ -3,7 +3,7 @@
 # Usage: ansible-playbook -i inventory/ plays/install-node-exporter.yml
 
 - name: Install Node Exporter
-  hosts: all
+  hosts: all:!pve_cluster
   become: yes
   gather_facts: yes
 

ansible/plays/install-ntp-ha.yml

@@ -22,54 +22,12 @@
 # so it stays BACKUP while the PVE nodes exist and wins the VIP by default once
 # they are gone -- no config change required at cutover.
 
-# Phase 1. Every static assignment of the IPv6 service address must be gone
-# fleet-wide before ANY host claims it via VRRP.
-#
-# DAD is enabled on the VRRP address (deliberately -- under single-master VRRP a
-# duplicate is a real fault worth detecting). So a static copy on one host and a
-# VRRP-managed copy on another cannot coexist: the kernel marks the newcomer
-# `dadfailed deprecated`, and its master then believes it is serving while
-# carrying an unusable address. That is not hypothetical -- it is what edge-1
-# did on 2026-08-07 when it took the address while three PVE nodes still held it
-# statically.
-#
-# Deliberately NOT serial: this play must complete everywhere before phase 2
-# starts anywhere. Doing the withdrawal per host inside phase 2 cannot avoid the
-# overlap, because with serial: 1 the first host claims the address while the
-# rest are still static.
-#
-# There is a gap between the phases where no host holds the address. It is
-# bounded by one play's run time and clients simply retry.
-- name: Withdraw static assignments of the NTP service address
-  hosts: ntp_cluster
-  become: true
-
-  vars:
-    keepalived_ntp_vip6: "{{ ntp_vip6 }}"
-    keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
-
-  tasks:
-    - name: Withdraw any static assignment of the service address
-      ansible.builtin.include_role:
-        name: keepalived-ntp
-        tasks_from: withdraw-static.yml
-      when: ntp_keepalived_enabled | default(false)
-
-# Phase 2. Configure chrony and hand the address to keepalived.
 - name: Install NTP Server (HA)
   hosts: ntp_cluster
   become: true
-  # One node at a time. The keepalived role withdraws the statically assigned
-  # IPv6 service address and restarts keepalived so the address is re-taken
-  # under VRRP control; doing that across every NTP server at once would leave
-  # the whole fleet with nobody holding fd40::123 until the elections settle.
-  # Serialised, the remaining members keep answering throughout.
-  serial: 1
 
   vars:
     ntp_vip: "10.2.40.4"
-    # ntp_vip6 / ntp_vip6_linklocal come from group_vars/ntp_cluster.yml, so
-    # both plays in this file see one definition.
     ntp_servers:
       jupiter:
         priority: 100
@@ -91,8 +49,6 @@
       when: ntp_keepalived_enabled | default(false)
       vars:
         keepalived_ntp_vip: "{{ ntp_vip }}"
-        keepalived_ntp_vip6: "{{ ntp_vip6 }}"
-        keepalived_ntp_vip6_linklocal: "{{ ntp_vip6_linklocal }}"
         keepalived_ntp_state: "{{ ntp_servers[inventory_hostname].state }}"
         keepalived_ntp_priority: "{{ ntp_servers[inventory_hostname].priority }}"
         keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
@@ -134,49 +90,6 @@
           {% endif %}
       when: ntp_keepalived_enabled | default(false)
 
-    # The v6 address is the path clients actually use, so it gets the same
-    # check as the v4 VIP. Without this a broken v6 rollout -- the address
-    # never claimed, or -nodad not behaving as expected on a given kernel --
-    # would not surface here at all, only later by reading
-    # /var/log/keepalived-state.log by hand.
-    #
-    # `ip -6 addr show ... to <addr>` prints the address when present and
-    # nothing when absent, so emptiness is the test.
-    #
-    # Presence cannot itself distinguish a VRRP-held address from a static
-    # leftover -- they look identical. It is only meaningful because the
-    # keepalived-ntp role asserts, on both distros, that no static assignment
-    # survives: the play fails before reaching here if one does.
-    - name: Check if this server holds the IPv6 service address
-      ansible.builtin.command:
-        argv:
-          - ip
-          - -6
-          - addr
-          - show
-          - dev
-          - "{{ ntp_keepalived_interface }}"
-          - to
-          - "{{ ntp_vip6 }}"
-      register: vip6_check
-      failed_when: false
-      changed_when: false
-      when: ntp_keepalived_enabled | default(false)
-
-    - name: Display IPv6 service address status
-      ansible.builtin.debug:
-        msg: >
-          {% if vip6_check.stdout | default('') | length > 0 %}
-          This server ({{ inventory_hostname }}) holds the IPv6 service address
-          {{ ntp_vip6 }} -- it is the v6 MASTER and is serving ntp.srv.renner.uno
-          {% else %}
-          This server ({{ inventory_hostname }}) does not hold {{ ntp_vip6 }}.
-          Expected on a v6 BACKUP. If NO host holds it, check_ntp is failing
-          everywhere -- inspect `chronyc tracking` and
-          /var/log/keepalived-ntp-health.log
-          {% endif %}
-      when: ntp_keepalived_enabled | default(false)
-
     - name: Report hosts serving NTP without participating in the VIP
       ansible.builtin.debug:
         msg: >

ansible/plays/install-owncloud.yml

@@ -0,0 +1,8 @@
+---
+- name: Install and Configure ownCloud Infinite Scale
+  hosts: proxmox_owncloud
+  become: true
+  roles:
+    - role: common
+    - role: docker-install
+    - role: owncloud

ansible/plays/install-pbs-client-synology.yml

@@ -0,0 +1,104 @@
+---
+# Synology-specific playbook using raw module (Python 3.8 workaround)
+- name: Install PBS Client on Synology NAS
+  hosts: synology_nas
+  gather_facts: false
+
+  vars:
+    pbs_client_data_dir: "/volume1/docker/pbs-client"
+
+  tasks:
+    - name: Create PBS client directory
+      ansible.builtin.raw: |
+        mkdir -p {{ pbs_client_data_dir }}/state
+      changed_when: true
+
+    - name: Create docker-compose.yml
+      ansible.builtin.raw: |
+        cat > {{ pbs_client_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
+        services:
+          pbs-client:
+            image: proxmox/proxmox-backup-client:latest
+            container_name: pbs-client
+            restart: unless-stopped
+            hostname: {{ pbs_backup_id }}
+            env_file:
+              - .env
+            volumes:
+              - {{ pbs_client_data_dir }}/state:/root/.proxmox-backup
+        {% for path in pbs_backup_paths %}
+              - "{{ path }}:{{ path }}:ro"
+        {% endfor %}
+            mem_limit: 2g
+            command: ["sleep", "infinity"]
+        COMPOSE_EOF
+      changed_when: true
+
+    - name: Create .env file
+      ansible.builtin.raw: |
+        cat > {{ pbs_client_data_dir }}/.env << 'ENV_EOF'
+        PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
+        PBS_PASSWORD={{ pbs_password }}
+        ENV_EOF
+        chmod 600 {{ pbs_client_data_dir }}/.env
+      changed_when: true
+
+    - name: Create backup script
+      ansible.builtin.raw: |
+        cat > {{ pbs_client_data_dir }}/backup.sh << 'SCRIPT_EOF'
+        #!/bin/bash
+        set -e
+        source {{ pbs_client_data_dir }}/.env
+
+        BACKUP_SOURCES=""
+        {% for path in pbs_backup_paths %}
+        BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
+        {% endfor %}
+
+        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
+        docker exec pbs-client proxmox-backup-client backup \
+            $BACKUP_SOURCES \
+            --backup-id {{ pbs_backup_id }} \
+            --repository "${PBS_REPOSITORY}"
+
+        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed, pruning..."
+        docker exec pbs-client proxmox-backup-client prune \
+            --repository "${PBS_REPOSITORY}" \
+            --backup-id {{ pbs_backup_id }} \
+            --keep-last {{ pbs_keep_last | default(3) }} \
+            --keep-daily {{ pbs_keep_daily | default(7) }} \
+            --keep-weekly {{ pbs_keep_weekly | default(4) }} \
+            --keep-monthly {{ pbs_keep_monthly | default(6) }}
+
+        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Done"
+        SCRIPT_EOF
+        chmod 755 {{ pbs_client_data_dir }}/backup.sh
+      changed_when: true
+
+    - name: Start PBS client container
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        cd {{ pbs_client_data_dir }} && docker compose pull && docker compose up -d
+      become: true
+      changed_when: true
+
+    - name: Setup cron job for daily backup at 2 AM (root crontab)
+      ansible.builtin.shell: |
+        # Remove existing entry if present
+        crontab -l 2>/dev/null | grep -v "pbs-client/backup.sh" | crontab - 2>/dev/null || true
+        # Add new entry
+        (crontab -l 2>/dev/null; echo "0 2 * * * {{ pbs_client_data_dir }}/backup.sh >> /var/log/pbs-backup.log 2>&1") | crontab -
+      become: true
+      changed_when: true
+
+    - name: Verify container is running
+      ansible.builtin.shell: |
+        export PATH=$PATH:/usr/local/bin
+        docker ps --filter name=pbs-client --format "{{ '{{' }}.Status{{ '}}' }}"
+      become: true
+      register: container_status
+      changed_when: false
+
+    - name: Show container status
+      ansible.builtin.debug:
+        msg: "PBS client container status: {{ container_status.stdout }}"

ansible/plays/install-pbs-client.yml

@@ -0,0 +1,11 @@
+---
+- name: Install PBS Client on Synology NAS
+  hosts: synology_nas
+  become: true
+  # Variables are set in:
+  # - Role defaults: ansible/roles/pbs-client/defaults/main.yml
+  # - Host overrides: ansible/inventory/host_vars/<hostname>/vars.yml
+  # - Secrets: ansible/inventory/host_vars/<hostname>/vault.yml
+
+  roles:
+    - pbs-client

ansible/plays/install-postgres-monitoring.yml

@@ -0,0 +1,62 @@
+---
+# Setup PostgreSQL monitoring user and exporter
+# Usage: ansible-playbook -i inventory/ plays/setup-postgres-monitoring.yml
+
+- name: Create PostgreSQL monitoring user on primary
+  hosts: authentik-db-1.srv.renner.uno
+  become: yes
+  vars_files:
+    - ../inventory/group_vars/all/vault.yml
+  tasks:
+    - name: Install psycopg2 for Ansible PostgreSQL modules
+      apt:
+        name: python3-psycopg2
+        state: present
+
+    - name: Create postgres_exporter user
+      become_user: postgres
+      community.postgresql.postgresql_user:
+        name: postgres_exporter
+        password: "{{ vault_postgres_exporter_password }}"
+        role_attr_flags: "NOSUPERUSER,NOCREATEDB,NOCREATEROLE"
+        state: present
+
+    - name: Grant pg_monitor role to exporter user
+      become_user: postgres
+      community.postgresql.postgresql_privs:
+        database: postgres
+        type: group
+        objs: pg_monitor
+        roles: postgres_exporter
+        grant_option: no
+
+    - name: Grant connect on authentik database
+      become_user: postgres
+      community.postgresql.postgresql_privs:
+        database: authentik
+        type: database
+        privs: CONNECT
+        roles: postgres_exporter
+
+- name: Deploy postgres-exporter to Prometheus server
+  hosts: prometheus.srv.renner.uno
+  become: yes
+  vars_files:
+    - ../inventory/group_vars/all/vault.yml
+  vars:
+    postgres_exporter_db_host: "10.2.40.135"  # PostgreSQL HA VIP
+    postgres_exporter_db_user: "postgres_exporter"
+    postgres_exporter_db_password: "{{ vault_postgres_exporter_password }}"
+    postgres_exporter_db_name: "postgres"
+  roles:
+    - postgres-exporter
+
+  post_tasks:
+    - name: Verify postgres-exporter is running
+      uri:
+        url: "http://127.0.0.1:9187/metrics"
+        status_code: 200
+      register: exporter_health
+      retries: 3
+      delay: 5
+      until: exporter_health.status == 200

ansible/plays/install-prometheus.yml

@@ -0,0 +1,43 @@
+---
+# Install Prometheus Metrics Server
+# Usage: ansible-playbook -i inventory/ plays/install-prometheus.yml
+
+- name: Install Prometheus Metrics Server
+  hosts: prometheus.srv.renner.uno
+  become: yes
+  gather_facts: yes
+
+  vars:
+    # Synology NAS SNMP targets
+    snmp_synology_targets:
+      - name: nas-1
+        target: "10.2.40.50"
+      - name: nas-2
+        target: "10.3.40.57"
+
+    # SNMPv3 credentials from vault
+    snmp_use_v3: true
+    snmp_v3_username: "{{ vault_snmp_v3_username }}"
+    snmp_v3_auth_password: "{{ vault_snmp_v3_auth_password }}"
+    snmp_v3_priv_password: "{{ vault_snmp_v3_priv_password }}"
+
+  roles:
+    - common
+    - prometheus-server
+    - snmp-exporter
+    - pve-exporter
+    - unifi-poller
+
+  post_tasks:
+    - name: Verify Prometheus is running
+      uri:
+        url: "http://127.0.0.1:9090/-/ready"
+        status_code: 200
+      register: prometheus_health
+      retries: 5
+      delay: 10
+      until: prometheus_health.status == 200
+
+    - name: Display Prometheus status
+      debug:
+        msg: "Prometheus is running and healthy at http://{{ inventory_hostname }}:9090"

ansible/plays/install-pve-monitoring.yml

@@ -0,0 +1,33 @@
+---
+# Deploy monitoring agents to Proxmox VE nodes
+# Usage: ansible-playbook -i inventory/ plays/setup-pve-monitoring.yml
+
+- name: Deploy Monitoring Agents to Proxmox Nodes
+  hosts: pve_cluster
+  become: yes
+  gather_facts: yes
+
+  vars:
+    # Monitoring endpoints
+    loki_endpoint: "http://loki.srv.renner.uno:3100"
+    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
+
+    # PVE-specific labels
+    alloy_extra_labels:
+      environment: "production"
+      datacenter: "homelab"
+      type: "hypervisor"
+
+  roles:
+    - monitoring-agent
+
+  post_tasks:
+    - name: Verify Alloy agent is running
+      systemd:
+        name: alloy
+        state: started
+      register: alloy_status
+
+    - name: Display agent status
+      debug:
+        msg: "Monitoring agent deployed to {{ inventory_hostname }}"

ansible/plays/install-uptime-kuma.yml

@@ -1,17 +0,0 @@
----
-# Deploy Uptime Kuma to the NetBird VPS
-# Usage: ansible-playbook -i inventory/ plays/install-uptime-kuma.yml
-#
-# Prerequisites:
-# - netbird-server play applied (docker, Traefik on netbird_netbird)
-# - OpenBao homelab/infrastructure/uptime-kuma with keys username, password
-# - OpenBao homelab/infrastructure/uptime-kuma/smtp with keys server, port, username, token
-# - homelab/infrastructure/uptime-kuma-metrics (key api_key) is written by the role
-# - VAULT_TOKEN in the environment (policy: uptime-kuma-vps)
-# - DNS for kuma.renner.uno pointing at the VPS
-
-- name: Deploy Uptime Kuma
-  hosts: netbird_server
-  become: true
-  roles:
-    - uptime-kuma

ansible/plays/install-vault.yml

@@ -0,0 +1,7 @@
+---
+- name: Install HashiCorp Vault
+  hosts: vault.srv.renner.uno
+  become: true
+  roles:
+    - common
+    - vault

ansible/plays/install-zwave-gateway.yml

@@ -1,15 +0,0 @@
----
-# Z-Wave gateway (zwave-js-ui) for hosts with the ZWA-2 USB radio attached.
-#
-# Targets [zwave_gateway]: membership means the radio is physically present.
-# Moving the stick to another edge host is a one-line inventory change.
-#
-# Requires the base play (plays/install-edge-host.yml) to have run: it needs
-# docker, and the flash-wear handling that the libeatmydata preload exists to
-# complement.
-- name: Install Z-Wave gateway
-  hosts: zwave_gateway
-  become: true
-
-  roles:
-    - zwave-gateway

ansible/plays/rotate-netbird-secrets.yml

@@ -1,250 +0,0 @@
----
-# Rotate the NetBird shared secrets that Ansible owns BOTH ends of.
-#
-# Deliberately opt-in. Rotation is not idempotent and it interrupts the mesh
-# briefly, so this play refuses to run unless invoked explicitly:
-#
-#   ansible-playbook plays/rotate-netbird-secrets.yml -e rotate_confirm=yes
-#
-# ONE-TIME MIGRATION REQUIRED BEFORE FIRST USE. The three keys below currently
-# live in group_vars/all/vault.yml. They must be deleted from there, because
-# group_vars files merge alphabetically and vault.yml sorts after
-# netbird-rotatable.yml — leaving them would let a later ordinary deploy
-# overwrite a freshly rotated secret with the old one. The play checks for this
-# and refuses to run until it is done:
-#
-#   ansible-vault edit inventory/group_vars/all/vault.yml
-#   # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
-#   #        and netbird_turn_secret
-#
-# That last one is easy to miss and breaks the rotation silently. vault.yml
-# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
-# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
-# a group var outranks a role default — so the bare key wins, and rotating only
-# the vault_-prefixed one changes nothing that reaches the template. The check
-# below matches both spellings for that reason.
-#
-# The first rotation then populates netbird-rotatable.yml with fresh values, so
-# nothing needs copying across.
-#
-# WHAT IS ROTATED, and why each is safe to rotate from here:
-#
-#   vault_netbird_turn_secret   coturn's `user=netbird:<secret>` in
-#                               turnserver.conf and the management server's
-#                               Turn.Secret/Password in management.json. Both
-#                               templates, one role, one host — the two ends
-#                               cannot disagree across a run.
-#   vault_netbird_relay_secret  management.json on the server and relay.env on
-#                               the proxies. Two hosts, both managed here.
-#
-# The test for "safe to rotate here" is whether the value appears in
-# management.json.j2. If the management server does not read it from a file
-# Ansible writes, Ansible does not own both ends, whatever else it templates.
-#
-# WHAT IS NOT, and must not be added:
-#
-#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on the
-#                                proxies, an Ansible template — but the
-#                                management server does not validate it from
-#                                any file. It is registered
-#                                in the management DATABASE. Rotating it changes
-#                                only what the proxies present, so every proxy
-#                                is locked out: "proxy auth failed: too many
-#                                failed authentication attempts", and the
-#                                proxies stop receiving their domain
-#                                configuration. Learned by doing it.
-#
-#   vault_netbird_datastore_key  encrypts NetBird's datastore at rest. Changing
-#                                it without re-encrypting the store makes the
-#                                existing data unreadable. Rotating this needs a
-#                                dump/restore, not a template rewrite.
-#   vault_netbird_auth_client_secret  the other half lives in Authentik. Ansible
-#                                only writes the consumer side, so rotating here
-#                                breaks login until Authentik is changed too.
-#
-# These are Ansible's OWN credentials, not a copy of the cluster's. OpenBao
-# carries credentials for some of the same systems (see
-# kubernetes/infrastructure/openbao/rotation/), and those are deliberately DIFFERENT
-# values: one consumer, one credential, independently revocable. Rotating here
-# must not touch OpenBao and does not invalidate anything the cluster holds —
-# that is the point of them being distinct, not drift to be reconciled.
-#
-# Corollary: if a credential ever has to be shared between the two, that is the
-# bug. Mint a second one instead.
-#
-# There IS a brief interruption. The server is reconfigured before the proxies,
-# so relay/proxy auth fails for the seconds between the two plays. Clients
-# reconnect; existing peer-to-peer tunnels are not torn down.
-
-- name: Generate and store new NetBird shared secrets
-  hosts: localhost
-  connection: local
-  gather_facts: false
-
-  vars:
-    rotatable_vault: "{{ playbook_dir }}/../inventory/group_vars/all/netbird-rotatable.yml"
-
-  pre_tasks:
-    - name: Require explicit confirmation
-      ansible.builtin.assert:
-        that:
-          - rotate_confirm | default('') == 'yes'
-        fail_msg: >-
-          Refusing to rotate without -e rotate_confirm=yes. This changes live
-          credentials and briefly interrupts the NetBird mesh.
-        success_msg: Confirmed; rotating NetBird shared secrets.
-
-    - name: Verify the vault password is available before touching anything
-      ansible.builtin.command:
-        cmd: "{{ playbook_dir }}/../vault-password.sh"
-      register: _vault_pw_check
-      changed_when: false
-      no_log: true
-
-    # Files inside group_vars/<group>/ are merged ALPHABETICALLY and the last
-    # one wins. netbird-rotatable.yml sorts before vault.yml, so if vault.yml
-    # still defines these keys it silently overrides everything this play
-    # writes. The rotation itself would still work — the plays below pass the
-    # new values explicitly, and play vars outrank group vars — but the NEXT
-    # ordinary run of install-netbird-server.yml or install-netbird-proxy.yml
-    # would read the stale value from vault.yml and push it back out, breaking
-    # server<->proxy auth. The deploy pipelines run unattended, so that would
-    # land on an unrelated change to those roles.
-    #
-    # Counting matches rather than printing them: the value never reaches
-    # stdout, only the key name does.
-    # Proving the vault is readable FIRST. Without this, a failed decrypt (wrong
-    # password, moved file) sends empty input to grep, which then reports zero
-    # matches — indistinguishable from a clean vault, so the guard below would
-    # wave the rotation through precisely when it knows least.
-    - name: Prove vault.yml is readable before trusting the key check
-      ansible.builtin.shell:
-        cmd: >-
-          set -o pipefail;
-          ansible-vault view
-          --vault-password-file {{ playbook_dir }}/../vault-password.sh
-          {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
-          | wc -l
-        executable: /bin/bash
-      register: _vault_lines
-      changed_when: false
-
-    - name: Check vault.yml does not still define the rotated keys
-      ansible.builtin.shell:
-        cmd: >-
-          ansible-vault view
-          --vault-password-file {{ playbook_dir }}/../vault-password.sh
-          {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
-          | grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
-      register: _stale_keys
-      changed_when: false
-      # grep exits 1 on "no matches", which is the good case here.
-      failed_when: false
-
-    - name: Refuse to rotate while vault.yml still owns these keys
-      ansible.builtin.assert:
-        that:
-          - _vault_lines.stdout | int > 0
-          - _stale_keys.stdout | int == 0
-        fail_msg: >-
-          vault.yml still defines {{ _stale_keys.stdout }} of the keys this play
-          rotates. group_vars files merge alphabetically and vault.yml sorts
-          last, so it would override netbird-rotatable.yml on the next deploy
-          and revert the secrets. Remove vault_netbird_turn_secret,
-          netbird_turn_secret and vault_netbird_relay_secret from vault.yml
-          first (one-time migration), then re-run.
-        success_msg: vault.yml does not shadow the rotated keys.
-
-  tasks:
-    # 48 alphanumerics. coturn's shared secret and the relay secret are both
-    # opaque strings; length is the only property that matters.
-    #
-    # ansible.builtin.password, NOT community.general.random_string: the CI
-    # image (ci/images/ansible/Containerfile) installs only community.proxmox
-    # and community.docker, so anything from community.general fails at this
-    # task when the play runs through the project's own tooling — even though
-    # it resolves fine on a workstation that happens to have the collection.
-    #
-    # `/dev/null` as the path makes the lookup generate without persisting;
-    # the value exists only in this fact.
-    - name: Generate replacement secrets
-      ansible.builtin.set_fact:
-        new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
-        new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
-      no_log: true
-
-    - name: Render the new values to a temporary file
-      ansible.builtin.copy:
-        dest: "{{ rotatable_vault }}.new"
-        mode: "0600"
-        content: |
-          ---
-          # Generated by plays/rotate-netbird-secrets.yml. Do not hand-edit —
-          # the next rotation overwrites this file wholesale.
-          vault_netbird_turn_secret: "{{ new_turn_secret }}"
-          vault_netbird_relay_secret: "{{ new_relay_secret }}"
-      no_log: true
-
-    # Encrypting the whole file rather than inline !vault strings: these three
-    # values are always rewritten together, so per-value encryption buys nothing
-    # and makes the write non-atomic.
-    - name: Encrypt the new file
-      ansible.builtin.command:
-        cmd: >-
-          ansible-vault encrypt
-          --vault-password-file {{ playbook_dir }}/../vault-password.sh
-          {{ rotatable_vault }}.new
-      changed_when: true
-
-    # Move into place only after encryption succeeded, so a failure above never
-    # leaves plaintext where the inventory would load it.
-    - name: Install the encrypted file
-      ansible.builtin.command:
-        cmd: "mv {{ rotatable_vault }}.new {{ rotatable_vault }}"
-      changed_when: true
-
-    - name: Confirm the installed file is encrypted
-      ansible.builtin.command:
-        cmd: "head -1 {{ rotatable_vault }}"
-      register: _vault_header
-      changed_when: false
-      failed_when: "'$ANSIBLE_VAULT' not in _vault_header.stdout"
-
-- name: Apply the new secrets to the NetBird server
-  hosts: netbird_server
-  become: true
-
-  vars:
-    netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
-    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-
-  roles:
-    - netbird-server
-
-- name: Apply the new secrets to the NetBird proxies
-  hosts: netbird_proxy
-  become: true
-  # One at a time so at least one relay stays reachable while the other
-  # restarts. The server is already on the new secret at this point, so a
-  # proxy is broken only between its own template write and its restart.
-  serial: 1
-
-  vars:
-    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-
-  roles:
-    - netbird-proxy
-
-- name: Report what still needs doing by hand
-  hosts: localhost
-  connection: local
-  gather_facts: false
-
-  tasks:
-    - name: Remaining manual steps
-      ansible.builtin.debug:
-        msg:
-          - "Rotated: turn_secret and relay_secret."
-          - "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
-          - "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
-          - "OpenBao is unaffected by design — its NetBird credentials are separate ones."

ansible/plays/update-authentik.yml

@@ -0,0 +1,10 @@
+---
+- name: Update Authentik HA
+  hosts: authentik_app
+  become: true
+  serial: 1
+  vars:
+    authentik_path: /home/ladmin
+
+  roles:
+    - authentik-update

ansible/plays/update-docker.yml

@@ -0,0 +1,13 @@
+---
+- name: Update Docker Containers
+  hosts: proxmox_docker
+  become: true
+  gather_facts: false
+
+  vars:
+    # Define the root directory where your docker-compose files live.
+    # You can override this per-host in host_vars if they differ.
+    docker_compose_dir: "/opt" 
+
+  roles:
+    - docker-update

ansible/plays/update-lxc.yml

@@ -0,0 +1,32 @@
+---
+- name: Update LXC Containers
+  hosts: proxmox_all_lxc:!proxmox_noansible
+  become: true
+  gather_facts: true
+  
+  tasks:
+    - name: Update APT Cache
+      ansible.builtin.apt:
+        update_cache: true
+        cache_valid_time: 3600
+      when: ansible_pkg_mgr == 'apt'
+
+    - name: Upgrade all packages
+      ansible.builtin.apt:
+        upgrade: dist
+      when: ansible_pkg_mgr == 'apt'
+      register: apt_upgrade
+
+    - name: Check if reboot required
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: reboot_required_file
+      when: ansible_pkg_mgr == 'apt'
+
+    - name: Reboot if required
+      ansible.builtin.reboot:
+        msg: "Rebooting due to system updates"
+      when: 
+        - ansible_pkg_mgr == 'apt'
+        - reboot_required_file.stat.exists
+        - apt_upgrade.changed

ansible/plays/update-pve-rolling.yml

@@ -0,0 +1,9 @@
+---
+- name: Proxmox VE Rolling Update
+  hosts: proxmox_nodes  # Replace with your inventory group name
+  become: true
+  serial: 1           # CRITICAL: Only update one node at a time
+  any_errors_fatal: true
+  
+  roles:
+    - pve-update

ansible/plays/update-vm.yml

@@ -0,0 +1,9 @@
+---
+- name: Update Proxmox VMs (Linux)
+  hosts: proxmox_all_qemu:&status_running:!proxmox_noansible:!proxmox_eol
+  become: true
+  serial: 1
+  gather_facts: false # We gather manually after checking connection
+
+  roles:
+    - vm-update

ansible/plays/update.yml

@@ -0,0 +1,28 @@
+---
+- name: Full Infrastructure Update
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Banner
+      debug:
+        msg: "Starting Full Infrastructure Update Sequence"
+
+- import_playbook: update-vm.yml
+- import_playbook: update-lxc.yml
+- import_playbook: update-authentik.yml
+# Re-render docker-compose templates before pulling updates
+# (immich decommissioned — migrated to in-cluster deployment, see kubernetes/apps/immich/)
+- import_playbook: install-n8n.yml
+- import_playbook: install-grist.yml
+- import_playbook: install-music-assistant.yml
+- import_playbook: update-docker.yml
+- import_playbook: install-log-aggregation.yml
+- import_playbook: update-pve-rolling.yml
+
+- name: Update Summary
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Done
+      debug:
+        msg: "Full Infrastructure Update Completed Successfully"

ansible/roles/authentik-stateless/defaults/main.yml

@@ -0,0 +1,18 @@
+---
+authentik_path: /home/ladmin
+authentik_db_host: "authentik-db.renner.uno"
+authentik_db_user: "authentik"
+authentik_db_name: "authentik"
+authentik_db_password: "changeMeInVault"
+authentik_secret_key: "changeMeInVault"
+authentik_tag: "latest" # or "2025.10.x"
+
+authentik_port_http: 9000
+authentik_port_https: 9443
+
+authentik_email_host: "smtp.example.com"
+authentik_email_port: 587
+authentik_email_user: "user"
+authentik_email_password: "password"
+authentik_email_use_tls: true
+authentik_email_from: "authentik@example.com"

ansible/roles/authentik-stateless/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: Restart Authentik
+  community.docker.docker_compose_v2: 
+    project_src: "{{ authentik_path }}"
+    state: restarted

ansible/roles/authentik-stateless/tasks/main.yml

@@ -0,0 +1,42 @@
+---
+- name: Ensure Authentik directory exists
+  ansible.builtin.file:
+    path: "{{ authentik_path }}"
+    state: directory
+    owner: "{{ ansible_user | default('root') }}"
+    group: "{{ ansible_user | default('root') }}"
+
+- name: Get latest Authentik release if no tag specified
+  ansible.builtin.uri:
+    url: https://api.github.com/repos/goauthentik/authentik/releases/latest
+    return_content: true
+  register: github_response
+  when: authentik_tag is not defined
+
+- name: Set target version fact
+  set_fact:
+    authentik_version: "{{ github_response.json.tag_name | regex_replace('^version/', '') if authentik_tag is not defined else authentik_tag }}"
+
+- name: Get Docker group ID
+  ansible.builtin.shell: "getent group docker | cut -d: -f3"
+  register: docker_gid_result
+  changed_when: false
+
+- name: Deploy .env configuration
+  ansible.builtin.template:
+    src: .env.j2
+    dest: "{{ authentik_path }}/.env"
+    mode: '0600'
+  notify: Restart Authentik
+
+- name: Deploy Docker Compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ authentik_path }}/docker-compose.yml"
+  notify: Restart Authentik
+
+- name: Ensure Authentik is started
+  community.docker.docker_compose_v2:
+    project_src: "{{ authentik_path }}"
+    state: present
+    pull: always

ansible/roles/authentik-stateless/templates/.env.j2

@@ -0,0 +1,38 @@
+PG_PASS={{ authentik_db_password }}
+PG_HOST={{ authentik_db_host }}
+PG_USER={{ authentik_db_user }}
+PG_DB={{ authentik_db_name }}
+AUTHENTIK_SECRET_KEY={{ authentik_secret_key }}
+AUTHENTIK_ERROR_REPORTING__ENABLED={{ authentik_error_reporting_enabled | default('true') }}
+AUTHENTIK_EMAIL__HOST={{ authentik_email_host }}
+AUTHENTIK_EMAIL__PORT={{ authentik_email_port }}
+AUTHENTIK_EMAIL__USERNAME={{ authentik_email_username }}
+AUTHENTIK_EMAIL__PASSWORD={{ authentik_email_password }}
+AUTHENTIK_EMAIL__USE_TLS={{ authentik_email_use_tls | lower }}
+AUTHENTIK_EMAIL__USE_SSL={{ authentik_email_use_ssl | lower }}
+AUTHENTIK_EMAIL__TIMEOUT={{ authentik_email_timeout }}
+AUTHENTIK_EMAIL__FROM={{ authentik_email_from }}
+AUTHENTIK_LOG_LEVEL={{ authentik_log_level | default('info') }}
+AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS={{ authentik_listen_trusted_proxy_cidrs | default('10.0.0.0/8,172.16.0.0/12') }}
+AUTHENTIK_TAG={{ authentik_tag | default(authentik_version) }}
+AUTHENTIK_ERROR_REPORTING__ENABLED=true
+AUTHENTIK_EMAIL__HOST={{ authentik_email_host }}
+AUTHENTIK_EMAIL__PORT={{ authentik_email_port }}
+AUTHENTIK_EMAIL__USERNAME={{ authentik_email_user }}
+AUTHENTIK_EMAIL__PASSWORD={{ authentik_email_password }}
+AUTHENTIK_EMAIL__USE_TLS={{ authentik_email_use_tls }}
+AUTHENTIK_EMAIL__FROM={{ authentik_email_from }}
+AUTHENTIK_PORT_HTTP={{ authentik_port_http }}
+AUTHENTIK_PORT_HTTPS={{ authentik_port_https }}
+AUTHENTIK_CSRF_TRUSTED_ORIGINS=https://authentik.renner.uno,https://auth.renner.uno,https://authentik-app-1.srv.renner.uno,https://authentik-app-2.srv.renner.uno
+
+# HA specific config:
+# Disable Redis (Use Postgres)
+AUTHENTIK_REDIS__HOST=""
+# Worker needs to use shared storage for media
+# Assuming nfs/glusterfs or just volume syncing?
+# Or maybe just use local volume if media uploads ar rare?
+# For true stateless, external S3 storage is recommended:
+# AUTHENTIK_STORAGE__MEDIA__BACKEND="aws"
+# AUTHENTIK_STORAGE__MEDIA__S3__BUCKET="..."
+# For now, local volume (will not sync across nodes without NFS).

ansible/roles/authentik-stateless/templates/docker-compose.yml.j2

@@ -0,0 +1,41 @@
+services:
+  server:
+    image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-latest}
+    restart: unless-stopped
+    command: server
+    env_file:
+      - .env
+    environment:
+      AUTHENTIK_REDIS__HOST: ""
+      AUTHENTIK_POSTGRESQL__HOST: ${PG_HOST}
+      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB}
+      AUTHENTIK_POSTGRESQL__USER: ${PG_USER}
+      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
+      AUTHENTIK_LISTEN__METRICS: "0.0.0.0:9300"
+    volumes:
+      - ./media:/media
+      - ./custom-templates:/templates
+      - ./certs:/certs
+    ports:
+      - "${AUTHENTIK_PORT_HTTP:-9000}:9000"
+      - "${AUTHENTIK_PORT_HTTPS:-9443}:9443"
+      - "9300:9300"
+
+  worker:
+    image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-latest}
+    restart: unless-stopped
+    command: worker
+    user: "1000:{{ docker_gid_result.stdout }}"
+    env_file:
+      - .env
+    environment:
+      AUTHENTIK_REDIS__HOST: ""
+      AUTHENTIK_POSTGRESQL__HOST: ${PG_HOST}
+      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB}
+      AUTHENTIK_POSTGRESQL__USER: ${PG_USER}
+      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./media:/media
+      - ./certs:/certs
+      - ./custom-templates:/templates

ansible/roles/authentik-update/tasks/main.yml

@@ -0,0 +1,47 @@
+---
+- name: Get latest Authentik release
+  ansible.builtin.uri:
+    url: https://api.github.com/repos/goauthentik/authentik/releases/latest
+    return_content: true
+  register: github_response
+
+- name: Set target version
+  ansible.builtin.set_fact:
+    target_version: "{{ github_response.json.tag_name | regex_replace('^version/', '') }}"
+
+- name: Display target version
+  ansible.builtin.debug:
+    msg: "Updating Authentik to version {{ target_version }}"
+
+- name: Update AUTHENTIK_TAG in .env
+  ansible.builtin.lineinfile:
+    path: "{{ authentik_path | default('/home/ladmin') }}/.env"
+    regexp: '^AUTHENTIK_TAG='
+    line: "AUTHENTIK_TAG={{ target_version }}"
+    create: true
+
+- name: Pull new images
+  ansible.builtin.command:
+    cmd: docker compose pull
+    chdir: "{{ authentik_path | default('/home/ladmin') }}"
+
+- name: Pull Outpost images
+  ansible.builtin.shell: |
+    docker pull ghcr.io/goauthentik/proxy:{{ target_version }}
+    docker pull ghcr.io/goauthentik/ldap:{{ target_version }}
+    docker pull ghcr.io/goauthentik/radius:{{ target_version }}
+  changed_when: true
+
+- name: Apply updates
+  ansible.builtin.command:
+    cmd: docker compose up -d --force-recreate
+    chdir: "{{ authentik_path | default('/home/ladmin') }}"
+
+- name: Restart worker to trigger outpost update
+  ansible.builtin.command: 
+    cmd: docker compose restart worker
+    chdir: "{{ authentik_path | default('/home/ladmin') }}"
+
+- name: Prune unused images
+  ansible.builtin.command:
+    cmd: docker image prune -f

ansible/roles/blackbox-exporter/defaults/main.yml

@@ -0,0 +1,26 @@
+---
+blackbox_exporter_version: "0.25.0"
+blackbox_exporter_port: 9115
+blackbox_exporter_user: "blackbox_exporter"
+blackbox_exporter_group: "blackbox_exporter"
+
+# Endpoints to probe
+blackbox_http_targets:
+  - name: "gitea"
+    url: "https://gitea.renner.uno"
+  - name: "immich"
+    url: "https://immich.renner.uno"
+  - name: "authentik"
+    url: "https://auth.renner.uno"
+  - name: "home-assistant"
+    url: "https://ha.renner.uno"
+  - name: "grafana"
+    url: "https://grafana.renner.uno"
+  - name: "prometheus"
+    url: "https://prometheus.renner.uno"
+
+blackbox_dns_targets: []
+
+blackbox_tcp_targets:
+  - name: "postgres-ha"
+    target: "10.2.40.135:5432"

ansible/roles/blackbox-exporter/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart blackbox_exporter
+  systemd:
+    name: blackbox_exporter
+    state: restarted
+    daemon_reload: yes

ansible/roles/blackbox-exporter/tasks/main.yml

@@ -0,0 +1,80 @@
+---
+- name: Create blackbox_exporter group
+  group:
+    name: "{{ blackbox_exporter_group }}"
+    system: yes
+    state: present
+
+- name: Create blackbox_exporter user
+  user:
+    name: "{{ blackbox_exporter_user }}"
+    group: "{{ blackbox_exporter_group }}"
+    system: yes
+    shell: /usr/sbin/nologin
+    createhome: no
+    state: present
+
+- name: Create blackbox_exporter directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    owner: "{{ blackbox_exporter_user }}"
+    group: "{{ blackbox_exporter_group }}"
+    mode: '0755'
+  loop:
+    - /etc/blackbox_exporter
+    - /var/lib/blackbox_exporter
+
+- name: Download blackbox_exporter
+  get_url:
+    url: "https://github.com/prometheus/blackbox_exporter/releases/download/v{{ blackbox_exporter_version }}/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64.tar.gz"
+    dest: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
+    mode: '0644'
+
+- name: Extract blackbox_exporter
+  unarchive:
+    src: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
+    dest: /tmp
+    remote_src: yes
+    creates: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64"
+
+- name: Install blackbox_exporter binary
+  copy:
+    src: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64/blackbox_exporter"
+    dest: /usr/local/bin/blackbox_exporter
+    owner: root
+    group: root
+    mode: '0755'
+    remote_src: yes
+  notify: Restart blackbox_exporter
+
+- name: Deploy blackbox_exporter configuration
+  template:
+    src: blackbox.yml.j2
+    dest: /etc/blackbox_exporter/blackbox.yml
+    owner: "{{ blackbox_exporter_user }}"
+    group: "{{ blackbox_exporter_group }}"
+    mode: '0644'
+  notify: Restart blackbox_exporter
+
+- name: Deploy blackbox_exporter systemd service
+  template:
+    src: blackbox_exporter.service.j2
+    dest: /etc/systemd/system/blackbox_exporter.service
+    mode: '0644'
+  notify: Restart blackbox_exporter
+
+- name: Enable and start blackbox_exporter
+  systemd:
+    name: blackbox_exporter
+    state: started
+    enabled: yes
+    daemon_reload: yes
+
+- name: Clean up downloaded files
+  file:
+    path: "{{ item }}"
+    state: absent
+  loop:
+    - "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
+    - "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64"

ansible/roles/blackbox-exporter/templates/blackbox.yml.j2

@@ -0,0 +1,60 @@
+modules:
+  http_2xx:
+    prober: http
+    timeout: 5s
+    http:
+      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
+      valid_status_codes: []  # Defaults to 2xx
+      method: GET
+      follow_redirects: true
+      fail_if_ssl: false
+      fail_if_not_ssl: false
+      tls_config:
+        insecure_skip_verify: false
+
+  http_post_2xx:
+    prober: http
+    timeout: 5s
+    http:
+      method: POST
+      valid_status_codes: []
+
+  tcp_connect:
+    prober: tcp
+    timeout: 5s
+
+  # DNS module for external domain resolution (tests upstream/cache)
+  dns_external:
+    prober: dns
+    timeout: 5s
+    dns:
+      query_name: "google.com"
+      query_type: "A"
+      valid_rcodes:
+        - NOERROR
+
+  # DNS module for internal domain resolution (tests local resolution)
+  dns_internal:
+    prober: dns
+    timeout: 5s
+    dns:
+      query_name: "k8s-1.srv.renner.uno"
+      query_type: "A"
+      valid_rcodes:
+        - NOERROR
+
+  # DNS module for common cached domain
+  dns_cached:
+    prober: dns
+    timeout: 5s
+    dns:
+      query_name: "cloudflare.com"
+      query_type: "A"
+      valid_rcodes:
+        - NOERROR
+
+  icmp:
+    prober: icmp
+    timeout: 5s
+    icmp:
+      preferred_ip_protocol: "ip4"

ansible/roles/blackbox-exporter/templates/blackbox_exporter.service.j2

@@ -0,0 +1,22 @@
+[Unit]
+Description=Prometheus Blackbox Exporter
+Documentation=https://github.com/prometheus/blackbox_exporter
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=simple
+User={{ blackbox_exporter_user }}
+Group={{ blackbox_exporter_group }}
+ExecStart=/usr/local/bin/blackbox_exporter \
+    --config.file=/etc/blackbox_exporter/blackbox.yml \
+    --web.listen-address=:{{ blackbox_exporter_port }}
+Restart=on-failure
+RestartSec=5s
+
+# Allow ICMP probing (requires CAP_NET_RAW)
+AmbientCapabilities=CAP_NET_RAW
+CapabilityBoundingSet=CAP_NET_RAW
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/chrony-ntp-server/defaults/main.yml

@@ -1,25 +1,9 @@
 ---
 # Chrony NTP Server Role Defaults
 
-# Networks allowed to use this server as NTP source.
-#
-# BOTH families are required. IPv6 is the primary address family here -- the
-# service address ntp.srv.renner.uno resolves to fd40::123 -- so an IPv4-only
-# allow list refuses every client that uses the documented name.
-#
-# The hosts carried `allow fd00::/8` for a long time without it ever being in
-# this file. Applying the role on 2026-08-07 rendered the template over it and
-# dropped the line on all four NTP servers at once, which took internal NTP
-# down over IPv6 until it was added here. Same class of untracked host state as
-# the anycast service address, and the same failure: config that exists only on
-# the box survives right up until something re-renders it.
+# Networks allowed to use this server as NTP source
 chrony_allowed_networks:
   - "10.2.0.0/16"
-  - "fd00::/8"
-  # Link-local. A client on the same segment may source its query from fe80::,
-  # and without this chrony refuses it -- the request arrives and is dropped,
-  # which looks like the server being unreachable rather than a policy refusal.
-  - "fe80::/10"
 
 # Primary NTP servers (PTB - Physikalisch-Technische Bundesanstalt, Stratum 1)
 chrony_upstream_servers:
@@ -34,26 +18,6 @@ chrony_upstream_pools:
 # Local stratum if disconnected from upstream
 chrony_local_stratum: 10
 
-# Step (rather than slew) the clock when the offset exceeds the threshold, but
-# only for the first N updates after chronyd starts. Without a makestep
-# directive chronyd only ever slews, so a large offset is never closed in
-# practice.
-#
-# edge-1, 2026-08-04: the CMOS battery is dead, so after a power loss the RTC
-# came up reading 2018-08-20. The kernel set the clock from it, systemd floored
-# that forward to its own built-in epoch (2026-06-26), and it ended up at
-# ~2026-07-19 -- about 16 days behind. chronyd then synced to ptbtime1, reported
-# "System time: 1376235.375 seconds slow of NTP time", and corrected none of it.
-# With host time before the cluster certificate's NotBefore, every outbound TLS
-# handshake failed x509 validation, taking out the MQTT bridge (heating
-# telemetry) and the Loki log shipper at once. The 16 days is incidental -- it
-# is just where the software flooring landed; the fault is that nothing steps.
-#
-# Limiting to the first few updates keeps steps confined to boot, so a
-# long-running server is never jumped out from under its workloads.
-chrony_makestep_threshold: 1.0
-chrony_makestep_limit: 3
-
 # Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
 # (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
 # error per interface when asked for it. Overridden to false in

ansible/roles/chrony-ntp-server/templates/chrony.conf.j2

@@ -14,11 +14,6 @@ pool {{ pool }}
 # Record the rate at which the system clock gains/losses time
 driftfile /var/lib/chrony/chrony.drift
 
-# Step the clock instead of slewing it when the offset is large, limited to the
-# first few updates after start so only boot-time corrections jump. Rationale
-# and the incident behind it: chrony_makestep_* in the role's defaults/main.yml.
-makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
-
 # Allow NTP client access from local networks
 {% for network in chrony_allowed_networks %}
 allow {{ network }}