fix(unifi): capture the API response on create only, not on every write

Root cause of the 60 permanently-dirty hairpin NAT rules.

The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.

No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:

  ignore_changes_to = ["_id"]                       plan 5661 — 60 remained
  ignore_all_server_changes = true                  plan 5662 — 60 remained
  restapi 3.x + ignore_server_additions = true      plan 5667 — 60 remained,
                                                    plus a new api_data diff

create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.

This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

.woodpecker/alert-monitor.yml

@@ -42,7 +42,7 @@ when:
 
 steps:
   - name: fetch-alerts
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/ci-manifests-test.yml

@@ -60,7 +60,7 @@ steps:
   # run that never touched them. Tracked in #461.
   - name: kustomize-build
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     commands:
       - |
         set -eu
@@ -115,7 +115,7 @@ steps:
   # An earlier revision of this comment also asserted that several deploy-*.yml
   # files emit "Additional property timeout is not allowed" warnings. That was
   # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # still hold on current CLIs — review raised the likelihood that the warning was
   # schema drift in a 7-minor-old CLI rather than real debt in those files, which
   # is one of the reasons this pin moved to latest stable. It cannot be confirmed
   # from a green run either, because the loop below sends lint output to /tmp/out
@@ -123,9 +123,15 @@ steps:
   # The claim is dropped rather than restated: what this step guarantees is the
   # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
+  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
+  # that, not a separate requirement — the server is bumped by image-automation
+  # within the 3.x range, so both should land on the same release.
+  #
+  # They do not land together, though: the server is a Kubernetes manifest and
+  # is bumped automatically, this pipeline file is not and cannot be (see
+  # infrastructure/image-automation/README.md). So the server moves first and
+  # this trails until someone bumps it by hand — which is what happened at
+  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
   #
   # It matters here because the CLI compiles the config against its own schema:
   # one behind the server can pass a config the server then rejects, or reject
@@ -152,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
     commands:
       - |
         set -eu
@@ -240,7 +246,7 @@ steps:
   # this step fails permanently on its own documentation.
   - name: no-digest-pins
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     commands:
       - |
         set -eu

.woodpecker/e2e-test.yml

@@ -4,7 +4,7 @@ when:
 
 steps:
   - name: e2e
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/e2e.yml

@@ -30,7 +30,7 @@ steps:
       - sh scripts/ci/test-model-selection.sh
 
   - name: e2e-test
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/issue-fix.yml

@@ -49,7 +49,7 @@ steps:
     # (the issue-pick logic needs jq, which only the ci-claude image has);
     # minting a 1h read-only token per run is cheap. Degrades to
     # snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

ansible/plays/install-blackbox-exporter.yml

@@ -1,19 +0,0 @@
----
-# Deploy Blackbox Exporter to Prometheus server for endpoint probing
-# Usage: ansible-playbook -i inventory/ plays/install-blackbox-exporter.yml
-
-- name: Install Blackbox Exporter
-  hosts: prometheus.srv.renner.uno
-  become: yes
-  roles:
-    - blackbox-exporter
-
-  post_tasks:
-    - name: Verify blackbox_exporter is running
-      uri:
-        url: "http://127.0.0.1:9115/metrics"
-        status_code: 200
-      register: exporter_health
-      retries: 3
-      delay: 5
-      until: exporter_health.status == 200

ansible/plays/install-log-aggregation.yml

@@ -1,86 +0,0 @@
----
-# Setup Log Aggregation Infrastructure
-# This playbook configures the complete logging stack:
-# 1. Loki server for log storage and querying
-# 2. Alloy agents on all VMs/CTs for log collection
-# 3. Syslog receiver for remote devices (Synology NAS, network devices)
-#
-# Usage: ansible-playbook -i inventory/ plays/setup-log-aggregation.yml
-#
-# To limit to specific hosts:
-#   --limit loki.srv.renner.uno          # Only configure Loki server
-#   --limit proxmox_all_running          # Only configure log agents
-
-- name: Setup Loki Log Aggregation Server
-  hosts: proxmox_logging
-  become: yes
-  gather_facts: yes
-
-  roles:
-    - common
-    - loki-server
-
-  post_tasks:
-    - name: Verify Loki is running
-      uri:
-        url: "http://127.0.0.1:3100/ready"
-        status_code: 200
-      register: loki_health
-      retries: 5
-      delay: 10
-      until: loki_health.status == 200
-
-    - name: Display Loki status
-      debug:
-        msg: "Loki is running and healthy at http://{{ inventory_hostname }}:3100"
-
-- name: Deploy Monitoring Agent to Loki Server (with syslog receiver)
-  hosts: proxmox_logging
-  become: yes
-  gather_facts: yes
-
-  vars:
-    loki_endpoint: "http://loki.srv.renner.uno:3100"
-    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
-    # Syslog receiver settings are defined in host_vars/loki.srv.renner.uno/vars.yml:
-    # - alloy_syslog_receiver_enabled: true
-    # - alloy_syslog_receiver_port: 1514
-    # - alloy_syslog_receiver_protocol: "udp"
-    # - alloy_syslog_receiver_format: "raw"
-    # - alloy_stability_level: "experimental"
-
-  roles:
-    - monitoring-agent
-
-  post_tasks:
-    - name: Display syslog receiver info
-      debug:
-        msg: |
-          Syslog receiver is listening on UDP port {{ alloy_syslog_receiver_port | default(1514) }}
-          Configure remote devices to send syslog to: {{ inventory_hostname }}:{{ alloy_syslog_receiver_port | default(1514) }}
-      when: alloy_syslog_receiver_enabled | default(false)
-
-- name: Deploy Monitoring Agent to All Running VMs/CTs
-  hosts: proxmox_all_running:!proxmox_logging:!proxmox_noansible
-  become: yes
-  gather_facts: yes
-  serial: 10  # Deploy to 10 hosts at a time to avoid overwhelming the network
-
-  vars:
-    loki_endpoint: "http://loki.srv.renner.uno:3100"
-    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
-
-  roles:
-    - monitoring-agent
-
-  post_tasks:
-    - name: Verify Alloy is running
-      systemd:
-        name: alloy
-        state: started
-      register: alloy_service
-
-    - name: Display agent status
-      debug:
-        msg: "Monitoring agent deployed successfully on {{ inventory_hostname }}"
-      when: alloy_service is defined

ansible/plays/install-loki.yml

@@ -1,26 +0,0 @@
----
-# Install Loki Log Aggregation Server
-# Usage: ansible-playbook -i inventory/ plays/install-loki.yml
-
-- name: Install Loki Log Aggregation Server
-  hosts: loki.srv.renner.uno
-  become: yes
-  gather_facts: yes
-
-  roles:
-    - common
-    - loki-server
-
-  post_tasks:
-    - name: Verify Loki is running
-      uri:
-        url: "http://127.0.0.1:3100/ready"
-        status_code: 200
-      register: loki_health
-      retries: 5
-      delay: 10
-      until: loki_health.status == 200
-
-    - name: Display Loki status
-      debug:
-        msg: "Loki is running and healthy at http://{{ inventory_hostname }}:3100"

ansible/plays/install-prometheus.yml

@@ -1,43 +0,0 @@
----
-# Install Prometheus Metrics Server
-# Usage: ansible-playbook -i inventory/ plays/install-prometheus.yml
-
-- name: Install Prometheus Metrics Server
-  hosts: prometheus.srv.renner.uno
-  become: yes
-  gather_facts: yes
-
-  vars:
-    # Synology NAS SNMP targets
-    snmp_synology_targets:
-      - name: nas-1
-        target: "10.2.40.50"
-      - name: nas-2
-        target: "10.3.40.57"
-
-    # SNMPv3 credentials from vault
-    snmp_use_v3: true
-    snmp_v3_username: "{{ vault_snmp_v3_username }}"
-    snmp_v3_auth_password: "{{ vault_snmp_v3_auth_password }}"
-    snmp_v3_priv_password: "{{ vault_snmp_v3_priv_password }}"
-
-  roles:
-    - common
-    - prometheus-server
-    - snmp-exporter
-    - pve-exporter
-    - unifi-poller
-
-  post_tasks:
-    - name: Verify Prometheus is running
-      uri:
-        url: "http://127.0.0.1:9090/-/ready"
-        status_code: 200
-      register: prometheus_health
-      retries: 5
-      delay: 10
-      until: prometheus_health.status == 200
-
-    - name: Display Prometheus status
-      debug:
-        msg: "Prometheus is running and healthy at http://{{ inventory_hostname }}:9090"

ansible/plays/update.yml

@@ -16,7 +16,6 @@
 - import_playbook: install-grist.yml
 - import_playbook: install-music-assistant.yml
 - import_playbook: update-docker.yml
-- import_playbook: install-log-aggregation.yml
 - import_playbook: update-pve-rolling.yml
 
 - name: Update Summary

ansible/roles/blackbox-exporter/defaults/main.yml

@@ -1,26 +0,0 @@
----
-blackbox_exporter_version: "0.25.0"
-blackbox_exporter_port: 9115
-blackbox_exporter_user: "blackbox_exporter"
-blackbox_exporter_group: "blackbox_exporter"
-
-# Endpoints to probe
-blackbox_http_targets:
-  - name: "gitea"
-    url: "https://gitea.renner.uno"
-  - name: "immich"
-    url: "https://immich.renner.uno"
-  - name: "authentik"
-    url: "https://auth.renner.uno"
-  - name: "home-assistant"
-    url: "https://ha.renner.uno"
-  - name: "grafana"
-    url: "https://grafana.renner.uno"
-  - name: "prometheus"
-    url: "https://prometheus.renner.uno"
-
-blackbox_dns_targets: []
-
-blackbox_tcp_targets:
-  - name: "postgres-ha"
-    target: "10.2.40.135:5432"

ansible/roles/blackbox-exporter/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart blackbox_exporter
-  systemd:
-    name: blackbox_exporter
-    state: restarted
-    daemon_reload: yes

ansible/roles/blackbox-exporter/tasks/main.yml

@@ -1,80 +0,0 @@
----
-- name: Create blackbox_exporter group
-  group:
-    name: "{{ blackbox_exporter_group }}"
-    system: yes
-    state: present
-
-- name: Create blackbox_exporter user
-  user:
-    name: "{{ blackbox_exporter_user }}"
-    group: "{{ blackbox_exporter_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    createhome: no
-    state: present
-
-- name: Create blackbox_exporter directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ blackbox_exporter_user }}"
-    group: "{{ blackbox_exporter_group }}"
-    mode: '0755'
-  loop:
-    - /etc/blackbox_exporter
-    - /var/lib/blackbox_exporter
-
-- name: Download blackbox_exporter
-  get_url:
-    url: "https://github.com/prometheus/blackbox_exporter/releases/download/v{{ blackbox_exporter_version }}/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64.tar.gz"
-    dest: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
-    mode: '0644'
-
-- name: Extract blackbox_exporter
-  unarchive:
-    src: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
-    dest: /tmp
-    remote_src: yes
-    creates: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64"
-
-- name: Install blackbox_exporter binary
-  copy:
-    src: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64/blackbox_exporter"
-    dest: /usr/local/bin/blackbox_exporter
-    owner: root
-    group: root
-    mode: '0755'
-    remote_src: yes
-  notify: Restart blackbox_exporter
-
-- name: Deploy blackbox_exporter configuration
-  template:
-    src: blackbox.yml.j2
-    dest: /etc/blackbox_exporter/blackbox.yml
-    owner: "{{ blackbox_exporter_user }}"
-    group: "{{ blackbox_exporter_group }}"
-    mode: '0644'
-  notify: Restart blackbox_exporter
-
-- name: Deploy blackbox_exporter systemd service
-  template:
-    src: blackbox_exporter.service.j2
-    dest: /etc/systemd/system/blackbox_exporter.service
-    mode: '0644'
-  notify: Restart blackbox_exporter
-
-- name: Enable and start blackbox_exporter
-  systemd:
-    name: blackbox_exporter
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
-    - "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64"

ansible/roles/blackbox-exporter/templates/blackbox.yml.j2

@@ -1,60 +0,0 @@
-modules:
-  http_2xx:
-    prober: http
-    timeout: 5s
-    http:
-      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
-      valid_status_codes: []  # Defaults to 2xx
-      method: GET
-      follow_redirects: true
-      fail_if_ssl: false
-      fail_if_not_ssl: false
-      tls_config:
-        insecure_skip_verify: false
-
-  http_post_2xx:
-    prober: http
-    timeout: 5s
-    http:
-      method: POST
-      valid_status_codes: []
-
-  tcp_connect:
-    prober: tcp
-    timeout: 5s
-
-  # DNS module for external domain resolution (tests upstream/cache)
-  dns_external:
-    prober: dns
-    timeout: 5s
-    dns:
-      query_name: "google.com"
-      query_type: "A"
-      valid_rcodes:
-        - NOERROR
-
-  # DNS module for internal domain resolution (tests local resolution)
-  dns_internal:
-    prober: dns
-    timeout: 5s
-    dns:
-      query_name: "k8s-1.srv.renner.uno"
-      query_type: "A"
-      valid_rcodes:
-        - NOERROR
-
-  # DNS module for common cached domain
-  dns_cached:
-    prober: dns
-    timeout: 5s
-    dns:
-      query_name: "cloudflare.com"
-      query_type: "A"
-      valid_rcodes:
-        - NOERROR
-
-  icmp:
-    prober: icmp
-    timeout: 5s
-    icmp:
-      preferred_ip_protocol: "ip4"

ansible/roles/blackbox-exporter/templates/blackbox_exporter.service.j2

@@ -1,22 +0,0 @@
-[Unit]
-Description=Prometheus Blackbox Exporter
-Documentation=https://github.com/prometheus/blackbox_exporter
-Wants=network-online.target
-After=network-online.target
-
-[Service]
-Type=simple
-User={{ blackbox_exporter_user }}
-Group={{ blackbox_exporter_group }}
-ExecStart=/usr/local/bin/blackbox_exporter \
-    --config.file=/etc/blackbox_exporter/blackbox.yml \
-    --web.listen-address=:{{ blackbox_exporter_port }}
-Restart=on-failure
-RestartSec=5s
-
-# Allow ICMP probing (requires CAP_NET_RAW)
-AmbientCapabilities=CAP_NET_RAW
-CapabilityBoundingSet=CAP_NET_RAW
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/heating-can/tasks/main.yml

@@ -34,17 +34,42 @@
 # stopped and masked rather than merely disabled -- masked so a stray
 # `systemctl start` or a leftover dependency cannot bring them back alongside
 # the containers.
-- name: Stop and mask the native CAN services
-  ansible.builtin.systemd:
-    name: "{{ item }}"
-    state: stopped
-    enabled: false
-    masked: true
+- name: Find which native CAN units still exist
+  ansible.builtin.stat:
+    path: "/etc/systemd/system/{{ item }}.service"
   loop:
     - open3e
     - e3oncan
-  failed_when: false
+  register: heating_can_native_units
 
+# Stop and disable BEFORE anything else, and do NOT swallow the result.
+#
+# The first version did stop + disable + mask in one call with
+# failed_when: false, then deleted the unit files. Masking a unit whose file
+# still exists conflicts, so the call failed, failed_when hid it, nothing was
+# stopped -- and the next task deleted the unit files out from under two
+# running processes. systemctl is-enabled then reported "not-found" and looked
+# clean while is-active still said "active".
+#
+# The result on 2026-08-10 was two open3e instances issuing UDS requests on the
+# same CAN bus for nine minutes, fighting over the open3e-edge client ID and
+# kicking each other off the broker.
+#
+# Only units that actually exist are touched, so a fresh host does not need an
+# error swallowed to get through this.
+- name: Stop and disable the native CAN services
+  ansible.builtin.systemd:
+    name: "{{ item.item }}"
+    state: stopped
+    enabled: false
+  loop: "{{ heating_can_native_units.results }}"
+  loop_control:
+    label: "{{ item.item }}"
+  when: item.stat.exists
+
+# Masking is deliberately NOT used. It writes a symlink at the same path the
+# next task removes, so the two cancel out -- which is half of how the original
+# failed. With the unit files gone there is nothing left to start.
 - name: Remove the native unit files
   ansible.builtin.file:
     path: "/etc/systemd/system/{{ item }}.service"
@@ -54,10 +79,30 @@
     - e3oncan
   notify: Reload systemd
 
-# The venvs are the point of this change -- leaving them would leave the thing
-# that churned. Their configuration is NOT removed: devices.json, the overlay
-# and the env file are still rendered below and bind-mounted into the
-# containers.
+# The invariant that actually matters is "no native process is on the CAN bus",
+# and systemd reporting success is not the same claim. Two open3e instances
+# polling one bus is the failure this whole block exists to prevent, so it is
+# asserted directly rather than inferred from the calls above.
+- name: Check for surviving native CAN processes
+  ansible.builtin.command:
+    argv: [pgrep, -af, "edge-host/(open3e|e3oncan)"]
+  register: heating_can_survivors
+  changed_when: false
+  failed_when: false
+
+- name: Assert no native CAN process survived
+  ansible.builtin.assert:
+    that:
+      - heating_can_survivors.stdout | default('') | length == 0
+    fail_msg: >-
+      A native open3e/e3oncan process is still running after the units were
+      stopped and removed:
+      {{ heating_can_survivors.stdout_lines | default([]) | join('; ') }}.
+      Starting the container now would put two readers on the same CAN bus --
+      two open3e instances issue competing UDS requests and take over each
+      other's MQTT session. Kill the survivors, then re-run.
+    success_msg: No native CAN processes remain; safe to start the container.
+
 - name: Remove the native venvs
   ansible.builtin.file:
     path: "{{ item }}"

ansible/roles/loki-server/defaults/main.yml

@@ -1,44 +0,0 @@
----
-# Loki Server Configuration
-
-# Version and download
-loki_version: "3.4.2"
-loki_arch: "linux-amd64"
-loki_download_url: "https://github.com/grafana/loki/releases/download/v{{ loki_version }}/loki-{{ loki_arch }}.zip"
-loki_bin_path: "/usr/local/bin/loki"
-
-# Directories
-loki_data_dir: "/var/lib/loki"
-loki_config_dir: "/etc/loki"
-loki_config_file: "{{ loki_config_dir }}/loki-config.yaml"
-
-# Server configuration
-loki_http_listen_port: 3100
-loki_grpc_listen_port: 9096
-loki_http_listen_address: "0.0.0.0"
-
-# Retention settings (90 days)
-loki_retention_period: "2160h"  # 90 days
-loki_retention_delete_delay: "2h"
-
-# Storage configuration
-loki_storage_type: "filesystem"
-loki_chunks_directory: "{{ loki_data_dir }}/chunks"
-loki_rules_directory: "{{ loki_data_dir }}/rules"
-
-# Index configuration
-loki_index_prefix: "loki_index_"
-loki_index_period: "24h"
-
-# Limits
-loki_ingestion_rate_mb: 10
-loki_ingestion_burst_size_mb: 20
-loki_max_streams_per_user: 10000
-loki_max_entries_limit_per_query: 5000
-
-# Query settings
-loki_max_query_length: "720h"  # 30 days max query span
-
-# Systemd service
-loki_user: "loki"
-loki_group: "loki"

ansible/roles/loki-server/handlers/main.yml

@@ -1,10 +0,0 @@
----
-- name: Restart loki
-  systemd:
-    name: loki
-    state: restarted
-    daemon_reload: yes
-
-- name: Reload systemd
-  systemd:
-    daemon_reload: yes

ansible/roles/loki-server/tasks/main.yml

@@ -1,95 +0,0 @@
----
-# Loki Server Installation and Configuration
-
-- name: Install dependencies
-  apt:
-    name:
-      - unzip
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create loki user
-  user:
-    name: "{{ loki_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ loki_user }}"
-    group: "{{ loki_group }}"
-    mode: '0755'
-  loop:
-    - "{{ loki_config_dir }}"
-    - "{{ loki_data_dir }}"
-    - "{{ loki_chunks_directory }}"
-    - "{{ loki_rules_directory }}"
-    - "{{ loki_data_dir }}/index"
-    - "{{ loki_data_dir }}/index_cache"
-    - "{{ loki_data_dir }}/compactor"
-
-- name: Download Loki
-  get_url:
-    url: "{{ loki_download_url }}"
-    dest: "/tmp/loki.zip"
-    mode: '0644'
-
-- name: Unzip Loki
-  unarchive:
-    src: "/tmp/loki.zip"
-    dest: "/tmp"
-    remote_src: yes
-
-- name: Install Loki binary
-  copy:
-    src: "/tmp/loki-{{ loki_arch }}"
-    dest: "{{ loki_bin_path }}"
-    mode: '0755'
-    remote_src: yes
-  notify: Restart loki
-
-- name: Create Loki configuration
-  template:
-    src: loki-config.yaml.j2
-    dest: "{{ loki_config_file }}"
-    owner: "{{ loki_user }}"
-    group: "{{ loki_group }}"
-    mode: '0644'
-  notify: Restart loki
-
-- name: Create Loki systemd service
-  template:
-    src: loki.service.j2
-    dest: /etc/systemd/system/loki.service
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart loki
-
-- name: Enable and start Loki service
-  systemd:
-    name: loki
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Wait for Loki to be ready
-  uri:
-    url: "http://127.0.0.1:{{ loki_http_listen_port }}/ready"
-    status_code: 200
-  register: result
-  until: result.status == 200
-  retries: 30
-  delay: 5
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/loki.zip"
-    - "/tmp/loki-{{ loki_arch }}"

ansible/roles/loki-server/templates/loki-config.yaml.j2

@@ -1,72 +0,0 @@
-# Loki Configuration
-# Generated by Ansible
-
-auth_enabled: false
-
-server:
-  http_listen_address: {{ loki_http_listen_address }}
-  http_listen_port: {{ loki_http_listen_port }}
-  grpc_listen_port: {{ loki_grpc_listen_port }}
-  log_level: info
-
-common:
-  instance_addr: 127.0.0.1
-  path_prefix: {{ loki_data_dir }}
-  storage:
-    filesystem:
-      chunks_directory: {{ loki_chunks_directory }}
-      rules_directory: {{ loki_rules_directory }}
-  replication_factor: 1
-  ring:
-    kvstore:
-      store: inmemory
-
-query_range:
-  results_cache:
-    cache:
-      embedded_cache:
-        enabled: true
-        max_size_mb: 100
-
-schema_config:
-  configs:
-    - from: 2024-01-01
-      store: tsdb
-      object_store: filesystem
-      schema: v13
-      index:
-        prefix: {{ loki_index_prefix }}
-        period: {{ loki_index_period }}
-
-storage_config:
-  tsdb_shipper:
-    active_index_directory: {{ loki_data_dir }}/index
-    cache_location: {{ loki_data_dir }}/index_cache
-
-compactor:
-  working_directory: {{ loki_data_dir }}/compactor
-  compaction_interval: 10m
-  retention_enabled: true
-  retention_delete_delay: {{ loki_retention_delete_delay }}
-  retention_delete_worker_count: 150
-  delete_request_store: filesystem
-
-limits_config:
-  retention_period: {{ loki_retention_period }}
-  ingestion_rate_mb: {{ loki_ingestion_rate_mb }}
-  ingestion_burst_size_mb: {{ loki_ingestion_burst_size_mb }}
-  max_streams_per_user: {{ loki_max_streams_per_user }}
-  max_entries_limit_per_query: {{ loki_max_entries_limit_per_query }}
-  max_query_length: {{ loki_max_query_length }}
-  reject_old_samples: true
-  reject_old_samples_max_age: 168h  # 7 days
-
-ruler:
-  alertmanager_url: ""
-  storage:
-    type: local
-    local:
-      directory: {{ loki_rules_directory }}
-
-analytics:
-  reporting_enabled: false

ansible/roles/loki-server/templates/loki.service.j2

@@ -1,25 +0,0 @@
-[Unit]
-Description=Loki Log Aggregation System
-Documentation=https://grafana.com/docs/loki/latest/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ loki_user }}
-Group={{ loki_group }}
-ExecStart={{ loki_bin_path }} -config.file={{ loki_config_file }}
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=on-failure
-RestartSec=10s
-LimitNOFILE=65536
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths={{ loki_data_dir }}
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/prometheus-server/defaults/main.yml

@@ -1,48 +0,0 @@
----
-# Prometheus Server Configuration
-
-# Version and download
-prometheus_version: "3.2.1"
-prometheus_arch: "linux-amd64"
-prometheus_download_url: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}.tar.gz"
-prometheus_bin_path: "/usr/local/bin/prometheus"
-prometheus_promtool_path: "/usr/local/bin/promtool"
-
-# Directories
-prometheus_data_dir: "/var/lib/prometheus"
-prometheus_config_dir: "/etc/prometheus"
-prometheus_config_file: "{{ prometheus_config_dir }}/prometheus.yml"
-prometheus_rules_dir: "{{ prometheus_config_dir }}/rules"
-prometheus_targets_dir: "{{ prometheus_config_dir }}/targets"
-
-# Server configuration
-prometheus_listen_address: "0.0.0.0"
-prometheus_listen_port: 9090
-prometheus_web_external_url: ""
-
-# Retention settings (90 days)
-prometheus_retention_time: "90d"
-prometheus_retention_size: "30GB"
-
-# Scrape settings
-prometheus_scrape_interval: "30s"
-prometheus_evaluation_interval: "30s"
-prometheus_scrape_timeout: "10s"
-
-# Remote write (for Alloy agents)
-prometheus_remote_write_enabled: true
-
-# Systemd service
-prometheus_user: "prometheus"
-prometheus_group: "prometheus"
-
-# Alert notification endpoints
-prometheus_alertmanager_url: ""
-
-# Email alerting (via Grafana)
-alerting_email_enabled: true
-alerting_email_to: ""
-
-# Home Assistant webhook for alerts
-alerting_ha_webhook_enabled: true
-alerting_ha_webhook_url: ""

ansible/roles/prometheus-server/handlers/main.yml

@@ -1,15 +0,0 @@
----
-- name: Restart prometheus
-  systemd:
-    name: prometheus
-    state: restarted
-    daemon_reload: yes
-
-- name: Reload prometheus
-  systemd:
-    name: prometheus
-    state: reloaded
-
-- name: Reload systemd
-  systemd:
-    daemon_reload: yes

ansible/roles/prometheus-server/tasks/main.yml

@@ -1,129 +0,0 @@
----
-# Prometheus Server Installation and Configuration
-
-- name: Install dependencies
-  apt:
-    name:
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create prometheus user
-  user:
-    name: "{{ prometheus_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0755'
-  loop:
-    - "{{ prometheus_config_dir }}"
-    - "{{ prometheus_data_dir }}"
-    - "{{ prometheus_rules_dir }}"
-    - "{{ prometheus_targets_dir }}"
-
-- name: Download Prometheus
-  get_url:
-    url: "{{ prometheus_download_url }}"
-    dest: "/tmp/prometheus.tar.gz"
-    mode: '0644'
-
-- name: Extract Prometheus
-  unarchive:
-    src: "/tmp/prometheus.tar.gz"
-    dest: "/tmp"
-    remote_src: yes
-
-- name: Install Prometheus binaries
-  copy:
-    src: "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}/{{ item.src }}"
-    dest: "{{ item.dest }}"
-    mode: '0755'
-    remote_src: yes
-  loop:
-    - { src: "prometheus", dest: "{{ prometheus_bin_path }}" }
-    - { src: "promtool", dest: "{{ prometheus_promtool_path }}" }
-  notify: Restart prometheus
-
-- name: Check for console templates
-  stat:
-    path: "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}/consoles"
-  register: consoles_dir
-
-- name: Copy console templates
-  copy:
-    src: "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}/{{ item }}/"
-    dest: "{{ prometheus_config_dir }}/{{ item }}/"
-    remote_src: yes
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-  loop:
-    - consoles
-    - console_libraries
-  when: consoles_dir.stat.exists
-
-- name: Create Prometheus configuration
-  template:
-    src: prometheus.yml.j2
-    dest: "{{ prometheus_config_file }}"
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0644'
-    validate: "{{ prometheus_promtool_path }} check config %s"
-  notify: Restart prometheus
-
-- name: Create alert rules
-  template:
-    src: alert-rules.yml.j2
-    dest: "{{ prometheus_rules_dir }}/alert-rules.yml"
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0644'
-  notify: Reload prometheus
-
-- name: Create node exporter targets file
-  template:
-    src: nodes.yml.j2
-    dest: "{{ prometheus_targets_dir }}/nodes.yml"
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0644'
-
-- name: Create Prometheus systemd service
-  template:
-    src: prometheus.service.j2
-    dest: /etc/systemd/system/prometheus.service
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart prometheus
-
-- name: Enable and start Prometheus service
-  systemd:
-    name: prometheus
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Wait for Prometheus to be ready
-  uri:
-    url: "http://127.0.0.1:{{ prometheus_listen_port }}/-/ready"
-    status_code: 200
-  register: result
-  until: result.status == 200
-  retries: 30
-  delay: 5
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/prometheus.tar.gz"
-    - "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}"

ansible/roles/prometheus-server/templates/alert-rules.yml.j2

@@ -1,100 +0,0 @@
-# Prometheus Alert Rules
-# Generated by Ansible
-
-groups:
-  - name: infrastructure
-    rules:
-      # Disk usage alerts
-      - alert: HighDiskUsage
-        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 < 15
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High disk usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Disk usage is above 85% on {{ '{{' }} $labels.mountpoint {{ '}}' }} ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}% free)"
-
-      - alert: CriticalDiskUsage
-        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 < 5
-        for: 2m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Critical disk usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Disk usage is above 95% on {{ '{{' }} $labels.mountpoint {{ '}}' }} ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}% free)"
-
-      # Memory alerts
-      - alert: HighMemoryUsage
-        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High memory usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Memory usage is above 90% ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}%)"
-
-      # CPU alerts
-      - alert: HighCPUUsage
-        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High CPU usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "CPU usage is above 85% for 10 minutes ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}%)"
-
-  - name: services
-    rules:
-      # Service down alerts
-      - alert: ServiceDown
-        expr: up == 0
-        for: 2m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Service {{ '{{' }} $labels.job {{ '}}' }} down on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Service has been unreachable for more than 2 minutes"
-
-      # DNS VIP health
-      - alert: DNSVIPDown
-        expr: absent(up{job="dns-servers"} == 1) or count(up{job="dns-servers"} == 1) < 2
-        for: 1m
-        labels:
-          severity: critical
-        annotations:
-          summary: "DNS cluster degraded"
-          description: "Less than 2 DNS servers are healthy"
-
-  - name: security
-    rules:
-      # High auth failure rate (placeholder - needs log-based metrics from Loki)
-      - alert: HighAuthFailureRate
-        expr: rate(authentik_login_failed_total[5m]) > 5
-        for: 2m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High authentication failure rate"
-          description: "More than 5 auth failures per minute detected"
-
-  - name: loki
-    rules:
-      # Loki health
-      - alert: LokiDown
-        expr: up{job="loki"} == 0
-        for: 2m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Loki log aggregation is down"
-          description: "Loki has been unreachable for more than 2 minutes"
-
-      # Loki ingestion rate
-      - alert: LokiHighIngestionRate
-        expr: rate(loki_distributor_bytes_received_total[5m]) > 10485760
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High log ingestion rate in Loki"
-          description: "Log ingestion rate is above 10MB/s"

ansible/roles/prometheus-server/templates/nodes.yml.j2

@@ -1,26 +0,0 @@
-# Node Exporter targets - auto-generated by Ansible
-# Updates require re-running the prometheus playbook
-#
-# NOTE: Most hosts have Alloy agents that push metrics via remote_write.
-# This file only includes hosts that need direct scraping (standalone node_exporter).
-# Alloy-managed hosts are excluded to avoid duplicate "down" targets.
-
-# PVE cluster nodes - need direct scraping (no Alloy agent)
-- targets:
-    - 'jupiter.renner.uno:9100'
-  labels:
-    job: 'node'
-    group: 'pve_cluster'
-    hostname: 'jupiter'
-- targets:
-    - 'neptune.renner.uno:9100'
-  labels:
-    job: 'node'
-    group: 'pve_cluster'
-    hostname: 'neptune'
-- targets:
-    - 'saturn.renner.uno:9100'
-  labels:
-    job: 'node'
-    group: 'pve_cluster'
-    hostname: 'saturn'

ansible/roles/prometheus-server/templates/prometheus.service.j2

@@ -1,38 +0,0 @@
-[Unit]
-Description=Prometheus Monitoring System
-Documentation=https://prometheus.io/docs/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ prometheus_user }}
-Group={{ prometheus_group }}
-ExecStart={{ prometheus_bin_path }} \
-  --config.file={{ prometheus_config_file }} \
-  --storage.tsdb.path={{ prometheus_data_dir }} \
-  --storage.tsdb.retention.time={{ prometheus_retention_time }} \
-  --storage.tsdb.retention.size={{ prometheus_retention_size }} \
-  --web.listen-address={{ prometheus_listen_address }}:{{ prometheus_listen_port }} \
-{% if prometheus_remote_write_enabled %}
-  --web.enable-remote-write-receiver \
-{% endif %}
-{% if prometheus_web_external_url %}
-  --web.external-url={{ prometheus_web_external_url }} \
-{% endif %}
-  --web.enable-lifecycle
-
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=on-failure
-RestartSec=10s
-LimitNOFILE=65536
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths={{ prometheus_data_dir }}
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/prometheus-server/templates/prometheus.yml.j2

@@ -1,304 +0,0 @@
-# Prometheus Configuration
-# Generated by Ansible
-
-global:
-  scrape_interval: {{ prometheus_scrape_interval }}
-  evaluation_interval: {{ prometheus_evaluation_interval }}
-  scrape_timeout: {{ prometheus_scrape_timeout }}
-  external_labels:
-    cluster: homelab
-    datacenter: renner
-
-rule_files:
-  - "{{ prometheus_rules_dir }}/*.yml"
-
-{% if prometheus_alertmanager_url %}
-alerting:
-  alertmanagers:
-    - static_configs:
-        - targets:
-            - {{ prometheus_alertmanager_url }}
-{% endif %}
-
-scrape_configs:
-  # Prometheus self-monitoring
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['localhost:{{ prometheus_listen_port }}']
-
-  # Loki metrics
-  - job_name: 'loki'
-    static_configs:
-      - targets: ['loki.srv.renner.uno:3100']
-
-  # File-based service discovery for dynamic targets
-  - job_name: 'alloy-nodes'
-    file_sd_configs:
-      - files:
-          - '{{ prometheus_targets_dir }}/*.yml'
-          - '{{ prometheus_targets_dir }}/*.json'
-        refresh_interval: 30s
-
-{% if prometheus_remote_write_enabled %}
-# Remote write receiver for Alloy agents
-# Agents push metrics to /api/v1/write
-{% endif %}
-
-# Static targets for infrastructure components
-  # Proxmox VE API metrics (via pve-exporter)
-  - job_name: 'pve'
-    scrape_interval: 60s
-    scrape_timeout: 55s
-    static_configs:
-      - targets:
-          - 'jupiter.renner.uno'
-          - 'neptune.renner.uno'
-          - 'saturn.renner.uno'
-    metrics_path: /pve
-    params:
-      module: [default]
-      cluster: ['1']
-      node: ['1']
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9221  # pve-exporter address
-      - target_label: group
-        replacement: proxmox
-      - target_label: type
-        replacement: hypervisor
-
-  # RustFS (S3-compatible storage)
-  # NOTE: RustFS uses OpenTelemetry push-model via --obs-endpoint, not Prometheus pull.
-  # Unlike MinIO, it doesn't expose /minio/v2/metrics/cluster endpoint.
-  # Metrics collection would require setting up RUSTFS_OBS_ENDPOINT to push to an OTEL collector.
-  # For now, node-level metrics are available via Alloy agents on the RustFS hosts.
-
-  # Home Assistant metrics
-  - job_name: 'homeassistant'
-    metrics_path: /api/prometheus
-    bearer_token: '{{ homeassistant_prometheus_token }}'
-    scheme: http
-    static_configs:
-      - targets: ['homeassistant.iot.renner.uno:8123']
-        labels:
-          group: 'smarthome'
-          type: 'homeassistant'
-
-  # UniFi Poller metrics
-  - job_name: 'unifi'
-    static_configs:
-      - targets: ['localhost:9130']
-        labels:
-          group: 'network'
-          type: 'unifi'
-
-  # Caddy reverse proxy metrics (HA cluster)
-  - job_name: 'caddy'
-    static_configs:
-      - targets:
-          - 'caddy.dmz.renner.uno:2019'
-          - 'caddy2.dmz.renner.uno:2019'
-        labels:
-          group: 'ingress'
-          type: 'reverse-proxy'
-
-  # PostgreSQL metrics (via postgres_exporter)
-  - job_name: 'postgresql'
-    static_configs:
-      - targets: ['localhost:9187']
-        labels:
-          group: 'database'
-          type: 'postgresql'
-
-  # Authentik identity provider metrics
-  - job_name: 'authentik'
-    static_configs:
-      - targets:
-          - 'authentik-app-1.srv.renner.uno:9300'
-          - 'authentik-app-2.srv.renner.uno:9300'
-        labels:
-          group: 'identity'
-          type: 'authentik'
-
-  # Node Exporter - system metrics from all hosts
-  - job_name: 'node'
-    file_sd_configs:
-      - files:
-          - '{{ prometheus_targets_dir }}/nodes.yml'
-        refresh_interval: 60s
-
-  # Chrony Exporter - NTP metrics from PVE cluster
-  # One static_config per host so each carries its own hostname label. The
-  # previous form listed all three targets together and recovered the hostname
-  # in relabel_configs by regex-matching __address__ against the node's IP --
-  # which silently coupled the labels to the addressing. Changing a target
-  # would have left the regex unmatched and the hostname label simply absent,
-  # with the scrape still succeeding.
-  - job_name: 'chrony'
-    static_configs:
-      - targets: ['jupiter.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'jupiter'
-      - targets: ['neptune.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'neptune'
-      - targets: ['saturn.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'saturn'
-
-  # Immich photo management metrics
-  - job_name: 'immich'
-    static_configs:
-      - targets: ['immich.srv.renner.uno:8081']
-        labels:
-          group: 'media'
-          type: 'photo-management'
-
-  # Blackbox Exporter - HTTP endpoint probing
-  - job_name: 'blackbox-http'
-    metrics_path: /probe
-    params:
-      module: [http_2xx]
-    static_configs:
-      - targets:
-          - https://gitea.renner.uno
-          - https://immich.renner.uno
-          - https://auth.renner.uno
-          - https://ha.renner.uno
-          - https://grafana.renner.uno
-          - https://prometheus.renner.uno
-        labels:
-          group: 'probe'
-          type: 'http'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - TCP probing
-  - job_name: 'blackbox-tcp'
-    metrics_path: /probe
-    params:
-      module: [tcp_connect]
-    static_configs:
-      - targets:
-          - 10.2.40.135:5432  # PostgreSQL HA VIP
-        labels:
-          group: 'probe'
-          type: 'tcp'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - DNS probing (external domains like google.com)
-  - job_name: 'blackbox-dns-external'
-    metrics_path: /probe
-    params:
-      module: [dns_external]
-    static_configs:
-      - targets:
-          - 10.2.40.53  # Proxmox DNS VIP
-          - 10.2.40.1   # UDM Gateway DNS
-        labels:
-          group: 'probe'
-          type: 'dns'
-          query_type: 'external'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - DNS probing (internal domains like k8s-1.srv.renner.uno)
-  - job_name: 'blackbox-dns-internal'
-    metrics_path: /probe
-    params:
-      module: [dns_internal]
-    static_configs:
-      - targets:
-          - 10.2.40.53  # Proxmox DNS VIP
-          - 10.2.40.1   # UDM Gateway DNS
-        labels:
-          group: 'probe'
-          type: 'dns'
-          query_type: 'internal'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - DNS probing (cached domains)
-  - job_name: 'blackbox-dns-cached'
-    metrics_path: /probe
-    params:
-      module: [dns_cached]
-    static_configs:
-      - targets:
-          - 10.2.40.53  # Proxmox DNS VIP
-          - 10.2.40.1   # UDM Gateway DNS
-        labels:
-          group: 'probe'
-          type: 'dns'
-          query_type: 'cached'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-{% if snmp_synology_targets is defined and snmp_synology_targets | length > 0 %}
-  # Synology NAS SNMP metrics
-  # snmp_exporter v0.27+ requires both synology and ucd_la_table modules
-  - job_name: 'synology'
-    scrape_interval: 60s
-    scrape_timeout: 55s
-    metrics_path: /snmp
-    params:
-      module: [synology, ucd_la_table, ucd_memory, if_mib]
-{% if snmp_use_v3 | default(false) %}
-      auth: [synology_v3]
-{% else %}
-      auth: [synology_v2]
-{% endif %}
-    static_configs:
-      - targets:
-{% for target in snmp_synology_targets %}
-          - '{{ target.target }}'  # {{ target.name }}
-{% endfor %}
-        labels:
-          group: 'storage'
-          type: 'nas'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__address__]
-        regex: '([^:]+)(:\d+)?'
-        replacement: '${1}'
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9116  # snmp_exporter address
-{% endif %}

ansible/roles/promtail/defaults/main.yml

@@ -1,36 +0,0 @@
----
-# Promtail Configuration
-
-# Version and download
-promtail_version: "3.4.2"
-promtail_arch: "linux-amd64"
-promtail_download_url: "https://github.com/grafana/loki/releases/download/v{{ promtail_version }}/promtail-{{ promtail_arch }}.zip"
-promtail_bin_path: "/usr/local/bin/promtail"
-
-# Directories
-promtail_config_dir: "/etc/promtail"
-promtail_config_file: "{{ promtail_config_dir }}/promtail-config.yaml"
-promtail_positions_file: "/var/lib/promtail/positions.yaml"
-
-# Server configuration
-promtail_http_listen_port: 9080
-promtail_http_listen_address: "0.0.0.0"
-
-# Loki server
-promtail_loki_url: "http://loki.srv.renner.uno:3100/loki/api/v1/push"
-
-# Log scrape configurations
-# Override in playbook or host_vars for specific services
-promtail_scrape_configs:
-  - job_name: system
-    static_configs:
-      - targets:
-          - localhost
-        labels:
-          job: syslog
-          host: "{{ inventory_hostname }}"
-          __path__: /var/log/syslog
-
-# Systemd service
-promtail_user: "promtail"
-promtail_group: "promtail"

ansible/roles/promtail/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart promtail
-  systemd:
-    name: promtail
-    state: restarted
-    daemon_reload: yes

ansible/roles/promtail/tasks/main.yml

@@ -1,112 +0,0 @@
----
-- name: Install dependencies
-  apt:
-    name:
-      - unzip
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create promtail group
-  group:
-    name: "{{ promtail_group }}"
-    system: yes
-    state: present
-
-- name: Create promtail user
-  user:
-    name: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    home: /var/lib/promtail
-    create_home: yes
-    state: present
-
-- name: Add promtail user to systemd-journal group
-  user:
-    name: "{{ promtail_user }}"
-    groups: systemd-journal
-    append: yes
-
-- name: Create config directory
-  file:
-    path: "{{ promtail_config_dir }}"
-    state: directory
-    owner: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    mode: '0755'
-
-- name: Create positions directory
-  file:
-    path: "{{ promtail_positions_file | dirname }}"
-    state: directory
-    owner: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    mode: '0755'
-
-- name: Check if promtail binary exists
-  stat:
-    path: "{{ promtail_bin_path }}"
-  register: promtail_binary
-
-- name: Get installed promtail version
-  command: "{{ promtail_bin_path }} --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: promtail_binary.stat.exists
-
-- name: Download promtail
-  get_url:
-    url: "{{ promtail_download_url }}"
-    dest: /tmp/promtail.zip
-    mode: '0644'
-  when: not promtail_binary.stat.exists or (installed_version.stdout is defined and promtail_version not in installed_version.stdout)
-
-- name: Extract promtail binary
-  unarchive:
-    src: /tmp/promtail.zip
-    dest: /tmp/
-    remote_src: yes
-  when: not promtail_binary.stat.exists or (installed_version.stdout is defined and promtail_version not in installed_version.stdout)
-
-- name: Install promtail binary
-  copy:
-    src: "/tmp/promtail-{{ promtail_arch }}"
-    dest: "{{ promtail_bin_path }}"
-    remote_src: yes
-    mode: '0755'
-  when: not promtail_binary.stat.exists or (installed_version.stdout is defined and promtail_version not in installed_version.stdout)
-  notify: Restart promtail
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - /tmp/promtail.zip
-    - "/tmp/promtail-{{ promtail_arch }}"
-
-- name: Configure promtail
-  template:
-    src: promtail-config.yaml.j2
-    dest: "{{ promtail_config_file }}"
-    owner: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    mode: '0644'
-  notify: Restart promtail
-
-- name: Create systemd service
-  template:
-    src: promtail.service.j2
-    dest: /etc/systemd/system/promtail.service
-    mode: '0644'
-  notify: Restart promtail
-
-- name: Enable and start promtail
-  systemd:
-    name: promtail
-    state: started
-    enabled: yes
-    daemon_reload: yes

ansible/roles/promtail/templates/promtail-config.yaml.j2

@@ -1,79 +0,0 @@
-server:
-  http_listen_port: {{ promtail_http_listen_port }}
-  http_listen_address: {{ promtail_http_listen_address }}
-  grpc_listen_port: 0
-
-positions:
-  filename: {{ promtail_positions_file }}
-
-clients:
-  - url: {{ promtail_loki_url }}
-    tenant_id: default
-
-scrape_configs:
-{% for config in promtail_scrape_configs %}
-  - job_name: {{ config.job_name }}
-{% if config.journal is defined %}
-    journal:
-      max_age: {{ config.journal.max_age | default('12h') }}
-      labels:
-        job: {{ config.job_name }}
-        host: {{ inventory_hostname }}
-{% if config.journal.matches is defined %}
-      matches: {{ config.journal.matches }}
-{% endif %}
-{% if config.relabel_configs is defined %}
-    relabel_configs:
-{% for relabel in config.relabel_configs %}
-      - source_labels: {{ relabel.source_labels | to_json }}
-        target_label: {{ relabel.target_label }}
-{% if relabel.regex is defined %}
-        regex: '{{ relabel.regex }}'
-{% endif %}
-{% if relabel.replacement is defined %}
-        replacement: '{{ relabel.replacement }}'
-{% endif %}
-{% endfor %}
-{% endif %}
-{% endif %}
-{% if config.static_configs is defined %}
-    static_configs:
-{% for static in config.static_configs %}
-      - targets:
-{% for target in static.targets %}
-          - {{ target }}
-{% endfor %}
-        labels:
-{% for key, value in static.labels.items() %}
-          {{ key }}: {{ value }}
-{% endfor %}
-{% endfor %}
-{% if config.pipeline_stages is defined %}
-    pipeline_stages:
-{% for stage in config.pipeline_stages %}
-{% if stage.multiline is defined %}
-      - multiline:
-          firstline: '{{ stage.multiline.firstline }}'
-{% if stage.multiline.max_wait_time is defined %}
-          max_wait_time: {{ stage.multiline.max_wait_time }}
-{% endif %}
-{% endif %}
-{% if stage.regex is defined %}
-      - regex:
-          expression: '{{ stage.regex.expression }}'
-{% endif %}
-{% if stage.labels is defined %}
-      - labels:
-{% for label in stage.labels %}
-          {{ label }}:
-{% endfor %}
-{% endif %}
-{% if stage.timestamp is defined %}
-      - timestamp:
-          source: {{ stage.timestamp.source }}
-          format: '{{ stage.timestamp.format }}'
-{% endif %}
-{% endfor %}
-{% endif %}
-{% endif %}
-{% endfor %}

ansible/roles/promtail/templates/promtail.service.j2

@@ -1,24 +0,0 @@
-[Unit]
-Description=Promtail Log Collector
-Documentation=https://grafana.com/docs/loki/latest/clients/promtail/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ promtail_user }}
-Group={{ promtail_group }}
-ExecStart={{ promtail_bin_path }} -config.file={{ promtail_config_file }}
-Restart=always
-RestartSec=5
-TimeoutStopSec=30
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths=/var/lib/promtail
-CapabilityBoundingSet=CAP_DAC_READ_SEARCH
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/pve-exporter/defaults/main.yml

@@ -1,19 +0,0 @@
----
-# PVE Exporter defaults
-
-# Installation
-pve_exporter_version: "3.4.5"
-pve_exporter_install_dir: /opt/pve-exporter
-pve_exporter_config_dir: /etc/pve-exporter
-pve_exporter_venv_path: "{{ pve_exporter_install_dir }}/venv"
-
-# Service
-pve_exporter_user: pve-exporter
-pve_exporter_group: pve-exporter
-pve_exporter_port: 9221
-
-# Proxmox connection (override with vault variables)
-pve_exporter_pve_user: "{{ pve_user | default('root@pam') }}"
-pve_exporter_pve_token_name: "{{ pve_token_id | default('ansible') }}"
-pve_exporter_pve_token_value: "{{ pve_token_secret }}"
-pve_exporter_pve_verify_ssl: false

ansible/roles/pve-exporter/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart pve-exporter
-  systemd:
-    name: pve-exporter
-    state: restarted
-    daemon_reload: yes

ansible/roles/pve-exporter/tasks/main.yml

@@ -1,78 +0,0 @@
----
-# PVE Exporter installation and configuration
-
-- name: Install Python dependencies
-  apt:
-    name:
-      - python3
-      - python3-pip
-      - python3-venv
-    state: present
-    update_cache: yes
-
-- name: Create pve-exporter system user
-  user:
-    name: "{{ pve_exporter_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create installation directory
-  file:
-    path: "{{ pve_exporter_install_dir }}"
-    state: directory
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    mode: '0755'
-
-- name: Create configuration directory
-  file:
-    path: "{{ pve_exporter_config_dir }}"
-    state: directory
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    mode: '0750'
-
-- name: Create Python virtual environment
-  command: python3 -m venv {{ pve_exporter_venv_path }}
-  args:
-    creates: "{{ pve_exporter_venv_path }}/bin/python"
-
-- name: Set venv ownership
-  file:
-    path: "{{ pve_exporter_venv_path }}"
-    state: directory
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    recurse: yes
-
-- name: Install prometheus-pve-exporter
-  pip:
-    name: prometheus-pve-exporter
-    virtualenv: "{{ pve_exporter_venv_path }}"
-    state: present
-
-- name: Create configuration file
-  template:
-    src: pve.yml.j2
-    dest: "{{ pve_exporter_config_dir }}/pve.yml"
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    mode: '0640'
-  notify: Restart pve-exporter
-
-- name: Create systemd service
-  template:
-    src: pve-exporter.service.j2
-    dest: /etc/systemd/system/pve-exporter.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify: Restart pve-exporter
-
-- name: Enable and start pve-exporter service
-  systemd:
-    name: pve-exporter
-    state: started
-    enabled: yes
-    daemon_reload: yes

ansible/roles/pve-exporter/templates/pve-exporter.service.j2

@@ -1,22 +0,0 @@
-[Unit]
-Description=Prometheus PVE Exporter
-Documentation=https://github.com/prometheus-pve/prometheus-pve-exporter
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ pve_exporter_user }}
-Group={{ pve_exporter_group }}
-ExecStart={{ pve_exporter_venv_path }}/bin/pve_exporter \
-  --config.file {{ pve_exporter_config_dir }}/pve.yml \
-  --web.listen-address 0.0.0.0:{{ pve_exporter_port }}
-Restart=on-failure
-RestartSec=10s
-
-# Security hardening
-NoNewPrivileges=yes
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/pve-exporter/templates/pve.yml.j2

@@ -1,8 +0,0 @@
-# PVE Exporter Configuration
-# Generated by Ansible
-
-default:
-  user: "{{ pve_exporter_pve_user }}"
-  token_name: "{{ pve_exporter_pve_token_name }}"
-  token_value: "{{ pve_exporter_pve_token_value }}"
-  verify_ssl: {{ pve_exporter_pve_verify_ssl | lower }}

ansible/roles/snmp-exporter/defaults/main.yml

@@ -1,33 +0,0 @@
----
-# SNMP Exporter defaults
-
-snmp_exporter_version: "0.30.1"
-snmp_exporter_listen_port: 9116
-snmp_exporter_user: "snmp_exporter"
-snmp_exporter_group: "snmp_exporter"
-
-snmp_exporter_install_dir: "/opt/snmp_exporter"
-snmp_exporter_config_dir: "/etc/snmp_exporter"
-
-# Synology NAS SNMP targets
-# Configure in host_vars or group_vars
-snmp_synology_targets: []
-# Example:
-# snmp_synology_targets:
-#   - name: nas-1
-#     target: 10.2.40.50
-#   - name: nas-2
-#     target: 10.2.40.51
-
-# SNMPv3 authentication (recommended for security)
-# These should be set in vault
-snmp_v3_username: "prometheus"
-snmp_v3_auth_protocol: "SHA"  # MD5 or SHA
-snmp_v3_auth_password: ""     # Set in vault
-snmp_v3_priv_protocol: "AES"  # DES or AES
-snmp_v3_priv_password: ""     # Set in vault
-snmp_v3_security_level: "authPriv"  # noAuthNoPriv, authNoPriv, authPriv
-
-# SNMPv2c fallback (less secure, but simpler)
-snmp_v2_community: "public"
-snmp_use_v3: true

ansible/roles/snmp-exporter/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart snmp_exporter
-  systemd:
-    name: snmp_exporter
-    state: restarted
-    daemon_reload: yes

ansible/roles/snmp-exporter/tasks/main.yml

@@ -1,114 +0,0 @@
----
-# SNMP Exporter installation and configuration
-
-- name: Create snmp_exporter group
-  group:
-    name: "{{ snmp_exporter_group }}"
-    system: yes
-    state: present
-
-- name: Create snmp_exporter user
-  user:
-    name: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    home: "{{ snmp_exporter_install_dir }}"
-    create_home: no
-
-- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    mode: '0755'
-  loop:
-    - "{{ snmp_exporter_install_dir }}"
-    - "{{ snmp_exporter_config_dir }}"
-
-- name: Check if snmp_exporter is already installed
-  stat:
-    path: "{{ snmp_exporter_install_dir }}/snmp_exporter"
-  register: snmp_exporter_binary
-
-- name: Get installed snmp_exporter version
-  command: "{{ snmp_exporter_install_dir }}/snmp_exporter --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: snmp_exporter_binary.stat.exists
-
-- name: Download snmp_exporter
-  get_url:
-    url: "https://github.com/prometheus/snmp_exporter/releases/download/v{{ snmp_exporter_version }}/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64.tar.gz"
-    dest: "/tmp/snmp_exporter.tar.gz"
-    mode: '0644'
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-
-- name: Extract snmp_exporter
-  unarchive:
-    src: "/tmp/snmp_exporter.tar.gz"
-    dest: "/tmp/"
-    remote_src: yes
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-
-- name: Install snmp_exporter binary
-  copy:
-    src: "/tmp/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64/snmp_exporter"
-    dest: "{{ snmp_exporter_install_dir }}/snmp_exporter"
-    remote_src: yes
-    owner: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    mode: '0755'
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-  notify: Restart snmp_exporter
-
-- name: Install default snmp.yml from release
-  copy:
-    src: "/tmp/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64/snmp.yml"
-    dest: "{{ snmp_exporter_config_dir }}/snmp.yml"
-    remote_src: yes
-    owner: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    mode: '0644'
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-  notify: Restart snmp_exporter
-
-- name: Add synology_v3 auth entry
-  lineinfile:
-    path: "{{ snmp_exporter_config_dir }}/snmp.yml"
-    insertafter: "^auths:"
-    line: "  synology_v3: {version: 3, security_level: {{ snmp_v3_security_level }}, username: {{ snmp_v3_username }}, password: {{ snmp_v3_auth_password }}, auth_protocol: {{ snmp_v3_auth_protocol }}, priv_protocol: {{ snmp_v3_priv_protocol }}, priv_password: {{ snmp_v3_priv_password }}}"
-  notify: Restart snmp_exporter
-
-- name: Add synology_v2 auth entry
-  lineinfile:
-    path: "{{ snmp_exporter_config_dir }}/snmp.yml"
-    insertafter: "^auths:"
-    line: "  synology_v2: {version: 2, community: {{ snmp_v2_community }}}"
-  notify: Restart snmp_exporter
-
-- name: Create snmp_exporter systemd service
-  template:
-    src: snmp_exporter.service.j2
-    dest: /etc/systemd/system/snmp_exporter.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify: Restart snmp_exporter
-
-- name: Enable and start snmp_exporter service
-  systemd:
-    name: snmp_exporter
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/snmp_exporter.tar.gz"
-    - "/tmp/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64"

ansible/roles/snmp-exporter/templates/snmp-auth.yml.j2

@@ -1,18 +0,0 @@
-# SNMPv3 Authentication for Synology NAS
-# Appended to default snmp.yml by Ansible
-
-auths:
-{% if snmp_use_v3 | default(true) %}
-  synology_v3:
-    security_level: {{ snmp_v3_security_level }}
-    username: {{ snmp_v3_username }}
-    password: {{ snmp_v3_auth_password }}
-    auth_protocol: {{ snmp_v3_auth_protocol }}
-{% if snmp_v3_security_level == 'authPriv' %}
-    priv_protocol: {{ snmp_v3_priv_protocol }}
-    priv_password: {{ snmp_v3_priv_password }}
-{% endif %}
-{% else %}
-  synology_v2:
-    community: {{ snmp_v2_community }}
-{% endif %}

ansible/roles/snmp-exporter/templates/snmp_exporter.service.j2

@@ -1,26 +0,0 @@
-[Unit]
-Description=Prometheus SNMP Exporter
-Documentation=https://github.com/prometheus/snmp_exporter
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ snmp_exporter_user }}
-Group={{ snmp_exporter_group }}
-ExecStart={{ snmp_exporter_install_dir }}/snmp_exporter \
-  --config.file={{ snmp_exporter_config_dir }}/snmp.yml \
-  --web.listen-address=:{{ snmp_exporter_listen_port }}
-
-Restart=on-failure
-RestartSec=5s
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-PrivateTmp=yes
-ReadOnlyPaths=/
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/unifi-poller/defaults/main.yml

@@ -1,34 +0,0 @@
----
-# UniFi Poller defaults
-
-# Version to install
-unifi_poller_version: "2.11.2"
-
-# Installation paths
-unifi_poller_bin_path: /usr/local/bin/unpoller
-unifi_poller_config_dir: /etc/unpoller
-unifi_poller_config_file: "{{ unifi_poller_config_dir }}/up.conf"
-
-# Service user
-unifi_poller_user: unpoller
-unifi_poller_group: unpoller
-
-# UniFi Controller settings (override in vars/vault)
-unifi_controller_url: "https://192.168.1.1"
-unifi_controller_user: ""
-unifi_controller_pass: ""
-unifi_controller_verify_ssl: false
-unifi_controller_site: "default"
-
-# Prometheus exporter settings
-unifi_poller_prometheus_port: 9130
-unifi_poller_prometheus_path: /metrics
-
-# Polling intervals
-unifi_poller_interval: 30s
-unifi_poller_save_sites: true
-unifi_poller_save_ids: false
-unifi_poller_save_events: false
-unifi_poller_save_alarms: false
-unifi_poller_save_anomalies: false
-unifi_poller_save_dpi: true

ansible/roles/unifi-poller/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart unpoller
-  systemd:
-    name: unpoller
-    state: restarted
-    daemon_reload: yes

ansible/roles/unifi-poller/tasks/main.yml

@@ -1,88 +0,0 @@
----
-# UniFi Poller installation and configuration
-
-- name: Create unpoller system user
-  user:
-    name: "{{ unifi_poller_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create configuration directory
-  file:
-    path: "{{ unifi_poller_config_dir }}"
-    state: directory
-    owner: "{{ unifi_poller_user }}"
-    group: "{{ unifi_poller_group }}"
-    mode: '0750'
-
-- name: Check if unpoller is already installed
-  stat:
-    path: "{{ unifi_poller_bin_path }}"
-  register: unpoller_binary
-
-- name: Get installed version
-  command: "{{ unifi_poller_bin_path }} --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: unpoller_binary.stat.exists
-
-- name: Download UniFi Poller
-  get_url:
-    url: "https://github.com/unpoller/unpoller/releases/download/v{{ unifi_poller_version }}/unpoller_{{ unifi_poller_version }}_linux_amd64.tar.gz"
-    dest: /tmp/unpoller.tar.gz
-    mode: '0644'
-  when: not unpoller_binary.stat.exists or (installed_version.stdout is defined and unifi_poller_version not in installed_version.stdout)
-
-- name: Extract UniFi Poller
-  unarchive:
-    src: /tmp/unpoller.tar.gz
-    dest: /tmp/
-    remote_src: yes
-  when: not unpoller_binary.stat.exists or (installed_version.stdout is defined and unifi_poller_version not in installed_version.stdout)
-
-- name: Install UniFi Poller binary
-  copy:
-    src: /tmp/unpoller
-    dest: "{{ unifi_poller_bin_path }}"
-    remote_src: yes
-    owner: root
-    group: root
-    mode: '0755'
-  when: not unpoller_binary.stat.exists or (installed_version.stdout is defined and unifi_poller_version not in installed_version.stdout)
-  notify: Restart unpoller
-
-- name: Create configuration file
-  template:
-    src: up.conf.j2
-    dest: "{{ unifi_poller_config_file }}"
-    owner: "{{ unifi_poller_user }}"
-    group: "{{ unifi_poller_group }}"
-    mode: '0640'
-  notify: Restart unpoller
-
-- name: Create systemd service
-  template:
-    src: unpoller.service.j2
-    dest: /etc/systemd/system/unpoller.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify: Restart unpoller
-
-- name: Enable and start unpoller service
-  systemd:
-    name: unpoller
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - /tmp/unpoller.tar.gz
-    - /tmp/unpoller
-  changed_when: false

ansible/roles/unifi-poller/templates/unpoller.service.j2

@@ -1,24 +0,0 @@
-[Unit]
-Description=UniFi Poller - Prometheus Exporter for UniFi Network
-Documentation=https://github.com/unpoller/unpoller
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ unifi_poller_user }}
-Group={{ unifi_poller_group }}
-ExecStart={{ unifi_poller_bin_path }} --config {{ unifi_poller_config_file }}
-Restart=on-failure
-RestartSec=10s
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadOnlyPaths=/
-ReadWritePaths={{ unifi_poller_config_dir }}
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/unifi-poller/templates/up.conf.j2

@@ -1,42 +0,0 @@
-# UniFi Poller Configuration
-# Generated by Ansible
-
-[poller]
-  # Turns on debug messages
-  debug = false
-  # Turns off messages
-  quiet = false
-
-[prometheus]
-  # Whether to export metrics
-  disable = false
-  # HTTP listen address for /metrics endpoint
-  http_listen = "0.0.0.0:{{ unifi_poller_prometheus_port }}"
-  # Path for metrics endpoint
-  report_errors = false
-
-[unifi]
-  # UniFi Controller configurations
-  [[unifi.controller]]
-    # URL for the UniFi Controller
-    url = "{{ unifi_controller_url }}"
-    # Username
-    user = "{{ unifi_controller_user }}"
-    # Password
-    pass = "{{ unifi_controller_pass }}"
-    # Sites to poll (leave empty for all sites)
-    sites = ["{{ unifi_controller_site }}"]
-    # Verify SSL certificates
-    verify_ssl = {{ unifi_controller_verify_ssl | lower }}
-    # Save site information
-    save_sites = {{ unifi_poller_save_sites | lower }}
-    # Save device IDs
-    save_ids = {{ unifi_poller_save_ids | lower }}
-    # Save events
-    save_events = {{ unifi_poller_save_events | lower }}
-    # Save alarms
-    save_alarms = {{ unifi_poller_save_alarms | lower }}
-    # Save anomalies
-    save_anomalies = {{ unifi_poller_save_anomalies | lower }}
-    # Save DPI data
-    save_dpi = {{ unifi_poller_save_dpi | lower }}

ci/images/heating-can/Containerfile

@@ -103,6 +103,8 @@ RUN open3e --help > /dev/null \
  && supervisord --help > /dev/null \
  && grep -q '^\[program:open3e\]' /etc/supervisord.conf \
  && grep -q '^\[program:e3oncan\]' /etc/supervisord.conf \
+ && grep -q '^\[supervisorctl\]' /etc/supervisord.conf \
+ && supervisorctl --help > /dev/null \
  && echo "self-test OK: both services execute; supervisord has both programs"
 
 # CAN_RAW needs no capabilities, so nothing here runs as root.

ci/images/heating-can/supervisord.conf

@@ -10,6 +10,24 @@
 # expands %(ENV_NAME)s at start, so neither the compose file nor a wrapper
 # script constructs open3e's or E3onCAN's argv -- those CLIs stay an
 # implementation detail of this image.
+# Control socket. Without these two sections supervisorctl cannot connect, so
+# there is no way to query or restart an individual service inside the
+# container -- which is the entire reason for running a supervisor rather than
+# one process per container. The first revision shipped without them.
+#
+# Unix socket only, inside the container: nothing outside needs to reach it,
+# and the container runs with host networking, so a TCP listener would be
+# exposed on the host.
+[unix_http_server]
+file=/tmp/supervisor.sock
+chmod=0700
+
+[rpcinterface:supervisor]
+supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
+
+[supervisorctl]
+serverurl=unix:///tmp/supervisor.sock
+
 [supervisord]
 nodaemon=true
 user=open3e

kubernetes/apps/authentik/blueprints-configmap.yaml

@@ -1326,7 +1326,121 @@ data:
           name: Caddy docs.renner.uno
         state: absent
 
-  # Passkey-only (WebAuthn) login for selected apps, in two parts:
+      # ---------------------------------------------------------------
+      # wordpress-gate — www.renner.uno
+      #
+      # Unlike every other gate here, this one does NOT front a whole
+      # host. www.renner.uno is a public family blog; only /wp-admin and
+      # /wp-login.php are routed to this outpost (see
+      # kubernetes/apps/wordpress/ingress.yaml). The front end never
+      # touches it.
+      #
+      # The reverse carve-out — gate everything, skip_path_regex the
+      # public parts, as paperless-gate does for /static/ — is not
+      # possible here: the public set is every post URL, so the pattern
+      # would need a negative lookahead and the outpost's RE2 has none.
+      - id: wordpress-authors-group
+        model: authentik_core.group
+        identifiers:
+          name: WordPress Authors
+        state: present
+        attrs:
+          is_superuser: false
+          # Git owns membership, as with the other gates: this group is the
+          # explicit allowlist for who may publish on www.renner.uno.
+          # Add/remove members HERE, not in the UI — blueprint apply is
+          # authoritative for this M2M and reverts UI edits.
+          #
+          # s.renner is the Authentik account for the site's WordPress
+          # Editor, which still carries the pre-2019 login `s.birzer` — the
+          # two names do not match, and that is expected.
+          users:
+            - !Find [authentik_core.user, [username, k.renner]]
+            - !Find [authentik_core.user, [username, s.renner]]
+
+      - id: wordpress-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: wordpress-gate
+        state: present
+        attrs:
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://wordpress.wordpress.svc.cluster.local
+          internal_host_ssl_validation: false
+          external_host: https://www.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+          # No skip_path_regex: wp-admin's assets are only ever fetched by an
+          # already-authenticated session, and wp-admin has no manifest or
+          # service worker — the credentials:'omit' PWA problem that forced
+          # carve-outs on vscode-gate and retired music-assistant-gate does
+          # not arise here.
+
+      - model: authentik_core.application
+        identifiers:
+          slug: wordpress-gate
+        state: present
+        attrs:
+          name: WordPress (Gate)
+          slug: wordpress-gate
+          provider: !KeyOf wordpress-gate-provider
+          meta_launch_url: https://www.renner.uno/wp-admin/
+          meta_description: "Forward-auth gate in front of wp-admin on www.renner.uno"
+
+      # Only WordPress Authors pass the gate — see the group above for the
+      # membership list. Both accounts that exist on the migrated site are
+      # covered, so the gate cannot lock out the editor the site is
+      # currently being kept alive for.
+      - model: authentik_policies.policybinding
+        identifiers:
+          target: !Find [authentik_core.application, [slug, wordpress-gate]]
+          group: !KeyOf wordpress-authors-group
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: wordpress-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf wordpress-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
+  # Passkey login for selected apps. Passkey is mandatory for accounts
+  # that hold one and unavailable to accounts that do not, so both flows
+  # branch on enrollment (policy passkey-user-has-webauthn): enrolled →
+  # WebAuthn, not enrolled → password + MFA. Enrollment status is a
+  # property of the identified account, so the weaker branch is never
+  # something a requester can select.
+  #
+  # Two parts:
   #
   # 1. A passwordless AUTHENTICATION flow (username → passkey, no
   #    password prompt) bound per-provider, so fresh logins to these
@@ -1335,7 +1449,13 @@ data:
   # 2. A step-up AUTHORIZATION flow — unlike the authentication flow,
   #    this runs on EVERY interactive authorize, so an existing
   #    password-only SSO session from another app cannot slip into
-  #    these apps without presenting a passkey.
+  #    these apps without presenting a passkey. Two things narrow that:
+  #    the stage is skipped outright for accounts with no passkey (they
+  #    have no way to satisfy it), and last_auth_threshold reduces it to
+  #    "without presenting a passkey in this browser within the window"
+  #    — one touch then covers every app, including a later
+  #    password-only session for the same user. The cookie is keyed to
+  #    that user's device, so it never carries across users.
   passkey-stepup.yaml: |
     version: 1
     metadata:
@@ -1356,43 +1476,99 @@ data:
           # flow this replaces.
           authentication: require_authenticated
 
-      - id: passkey-stepup-webauthn-validation
+      # ONE stage object, shared by the step-up authorization flow and
+      # the passwordless authentication flow below.
+      #
+      # `last_auth_threshold` is enforced with a single browser cookie
+      # (`authentik_mfa`) whose JWT is both signed with and carries the
+      # stage's PK, so a validation stage only ever honours a cookie it
+      # wrote itself. Two separate stage objects therefore each demand
+      # their own touch, and — sharing the one cookie name — overwrite
+      # each other, so neither window survives. Sharing this object is
+      # what lets the login touch satisfy the per-app step-up.
+      #
+      # Declared here, ahead of every !KeyOf reference to it: the
+      # blueprint importer resolves entries top-to-bottom.
+      - id: passkey-webauthn-validation
         model: authentik_stages_authenticator_validate.authenticatorvalidatestage
         identifiers:
-          name: passkey-stepup-webauthn-validation
+          name: passkey-webauthn-validation
         state: present
         attrs:
           device_classes:
             - webauthn
-          # `configure`, not `deny`: g.renner has Home Assistant access
-          # but no enrolled WebAuthn device (live check 2026-07-07), so
-          # a hard deny would lock her out with no self-service path.
-          # `configure` injects the default WebAuthn enrollment stage
-          # inline — first login without a passkey becomes "enroll one
-          # now, then validate it". Still passkey-only: there is no way
-          # through this stage without a WebAuthn credential.
-          not_configured_action: configure
-          configuration_stages:
-            - !Find [authentik_stages_authenticator_webauthn.authenticatorwebauthnstage, [name, default-authenticator-webauthn-setup]]
-          # Skip re-validation if a passkey was already presented within
-          # the window. Non-zero for two reasons: (1) no double-touch
-          # when the login itself used a passkey seconds earlier;
-          # (2) NetBird's dashboard silently re-authorizes via
-          # /silent-auth (prompt=none) on token expiry — a flow that
-          # always demands interaction would fail those and force a
-          # visible re-login every access-token lifetime.
+          # `deny`, not `configure`: the authentication flow reaches this
+          # stage with the user merely *identified* (a typed username)
+          # and no password stage in front of it, so inline enrollment
+          # would let anyone who knows a passkey-less username enroll
+          # their own passkey and take over that account. The policy
+          # binding means only accounts that already hold a passkey reach
+          # this stage at all, so `deny` is defence in depth rather than
+          # a path anyone walks — accounts without one branch to password
+          # + MFA instead.
+          not_configured_action: deny
+          # Window within which an already-presented passkey satisfies a
+          # later run of this stage. Non-zero for two reasons: (1) the
+          # per-app step-up must not demand a second touch seconds after
+          # the login touch; (2) NetBird's dashboard silently
+          # re-authorizes via /silent-auth (prompt=none) on token expiry —
+          # a stage that always demands interaction fails those and
+          # forces a visible re-login every access-token lifetime.
+          #
+          # The cookie is browser-scoped, so inside the window a re-login
+          # in that same browser is satisfied by the cookie alone and the
+          # username is all that gets typed. Accepted for a single touch
+          # per app.
           last_auth_threshold: hours=8
           webauthn_user_verification: preferred
 
-      - model: authentik_flows.flowstagebinding
+      # Gate on "this account has a passkey", so enrollment status picks
+      # the branch. Enrolled accounts must use WebAuthn — at login and at
+      # every authorize — while accounts with no passkey fall back to
+      # password + MFA. An attacker cannot pick the weaker branch: the
+      # user is already identified when this runs, so the branch follows
+      # the victim's enrollment, not the requester's choice.
+      #
+      # request.user is the flow's pending_user, not the (anonymous)
+      # requester: ReevaluateMarker builds the engine with
+      # plan.context[PLAN_CONTEXT_PENDING_USER] (flows/markers.py). That
+      # requires the bindings below to keep evaluate_on_plan false and
+      # re_evaluate_policies true — on-plan evaluation happens before the
+      # identification stage has resolved a user.
+      - id: policy-has-passkey
+        model: authentik_policies_expression.expressionpolicy
+        identifiers:
+          name: passkey-user-has-webauthn
+        state: present
+        attrs:
+          expression: |
+            return ak_user_has_authenticator(request.user, "webauthn")
+
+      - id: binding-stepup-webauthn
+        model: authentik_flows.flowstagebinding
         identifiers:
           target: !KeyOf passkey-stepup-flow
-          stage: !KeyOf passkey-stepup-webauthn-validation
+          stage: !KeyOf passkey-webauthn-validation
           order: 10
         state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-stepup-webauthn
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
 
       # Passwordless authentication flow: identification (username or
-      # email, NO password stage) → WebAuthn validation → login.
+      # email, NO password stage) → WebAuthn validation → login, with a
+      # password + MFA branch for accounts that have no passkey.
       - id: passkey-passwordless-flow
         model: authentik_flows.flow
         identifiers:
@@ -1403,37 +1579,6 @@ data:
           title: "Sign in with your passkey"
           designation: authentication
 
-      # SEPARATE stage from the step-up one, and deliberately `deny`:
-      # at this point in an authentication flow the user has only been
-      # *identified* (typed a username), not authenticated — `configure`
-      # here would let anyone who knows a passkey-less username enroll
-      # their own passkey and take over that account. Inline enrollment
-      # for passkey-less users happens exclusively in the authorization
-      # step-up stage above, where require_authenticated guarantees a
-      # fully authenticated session. A passkey-less user therefore
-      # bootstraps via password login on auth.renner.uno, then gets the
-      # enrollment prompt on first app access.
-      #
-      # Declared BEFORE the identification stage because the latter
-      # references it via !KeyOf (webauthn_stage), and the blueprint
-      # importer resolves entries top-to-bottom.
-      - id: passkey-passwordless-webauthn-validation
-        model: authentik_stages_authenticator_validate.authenticatorvalidatestage
-        identifiers:
-          name: passkey-passwordless-webauthn-validation
-        state: present
-        attrs:
-          device_classes:
-            - webauthn
-          not_configured_action: deny
-          # Small non-zero window so a passkey presented via the
-          # identification stage's autofill (webauthn_stage below) is
-          # not challenged a second time by this stage at order 20 in
-          # the same flow run. The skip is session-scoped, so a fresh
-          # (attacker) session never inherits it.
-          last_auth_threshold: minutes=5
-          webauthn_user_verification: preferred
-
       # webauthn_stage enables passkey autofill (conditional WebAuthn,
       # authentik >= 2025.12): the browser offers enrolled passkeys in
       # the username field's autofill — one tap, no username typed.
@@ -1449,7 +1594,7 @@ data:
           user_fields:
             - username
             - email
-          webauthn_stage: !KeyOf passkey-passwordless-webauthn-validation
+          webauthn_stage: !KeyOf passkey-webauthn-validation
 
       - model: authentik_flows.flowstagebinding
         identifiers:
@@ -1458,12 +1603,87 @@ data:
           order: 10
         state: present
 
-      - model: authentik_flows.flowstagebinding
+      # Order 20 and 21/22 are the two mutually exclusive branches.
+      - id: binding-passwordless-webauthn
+        model: authentik_flows.flowstagebinding
         identifiers:
           target: !KeyOf passkey-passwordless-flow
-          stage: !KeyOf passkey-passwordless-webauthn-validation
+          stage: !KeyOf passkey-webauthn-validation
           order: 20
         state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-passwordless-webauthn
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
+
+      # The negated branch. failure_result: true is the safety net — the
+      # UserLoginStage defaults a missing authentication backend to
+      # BACKEND_INBUILT instead of refusing (user_login/stage.py), so a
+      # flow that drops BOTH branches would log the user in on a typed
+      # username alone. A PolicyException is resolved to failure_result
+      # before negate is applied (policies/process.py), so a broken
+      # expression pulls the password stage IN and degrades everyone to
+      # password + MFA rather than to no factor at all.
+      - id: binding-passwordless-password
+        model: authentik_flows.flowstagebinding
+        identifiers:
+          target: !KeyOf passkey-passwordless-flow
+          stage: !Find [authentik_stages_password.passwordstage, [name, default-authentication-password]]
+          order: 21
+        state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-passwordless-password
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: true
+          failure_result: true
+
+      # Same MFA validation the default authentication flow runs at its
+      # order 30, so the fallback is not a weaker way into these apps
+      # than auth.renner.uno's own login already is. It carries
+      # not_configured_action: configure, which is safe here and nowhere
+      # else in this flow: the password stage above has established a
+      # first factor by the time it runs, so it doubles as the enrollment
+      # path for accounts holding no authenticator at all.
+      - id: binding-passwordless-mfa
+        model: authentik_flows.flowstagebinding
+        identifiers:
+          target: !KeyOf passkey-passwordless-flow
+          stage: !Find [authentik_stages_authenticator_validate.authenticatorvalidatestage, [name, default-authentication-mfa-validation]]
+          order: 22
+        state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-passwordless-mfa
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: true
+          failure_result: true
 
       - model: authentik_flows.flowstagebinding
         identifiers:
@@ -1577,3 +1797,19 @@ data:
           redirect_uris:
             - url: https://dash.renner.uno/login/generic_oauth
               matching_mode: strict
+
+      # Superseded by the single passkey-webauthn-validation stage above.
+      # Their flowstagebindings are FK-CASCADEd away with them; the
+      # identification stage's webauthn_stage FK is SET_NULL and is
+      # re-pointed above, so no dangling reference survives. Declared
+      # last: the importer applies entries top-to-bottom inside one
+      # transaction, so the replacements exist before these go.
+      - model: authentik_stages_authenticator_validate.authenticatorvalidatestage
+        identifiers:
+          name: passkey-stepup-webauthn-validation
+        state: absent
+
+      - model: authentik_stages_authenticator_validate.authenticatorvalidatestage
+        identifiers:
+          name: passkey-passwordless-webauthn-validation
+        state: absent

kubernetes/apps/authentik/kustomization.yaml

@@ -14,6 +14,7 @@ resources:
   - zone-configurator-gate-egress.yaml
   - alertmanager-gate-egress.yaml
   - prometheus-gate-egress.yaml
+  - wordpress-gate-egress.yaml
   - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml

kubernetes/apps/authentik/wordpress-gate-egress.yaml

@@ -0,0 +1,28 @@
+---
+# Allow the dedicated wordpress-gate outpost to egress into the wordpress
+# namespace and reach WordPress (:80). Without this, the authentik-namespace
+# baseline CNP has no egress rule to wordpress, so the gate outpost's
+# forwarded requests are dropped. Scoped to the gate outpost's pod label so no
+# other authentik component can reach it. Mirrors esphome-gate-egress.yaml.
+#
+# Note this gate only ever sees /wp-admin and /wp-login.php — the public site
+# is routed straight to the same Service by a separate Ingress, so this rule
+# is not the only path into the wordpress namespace, unlike the paperless case.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: wordpress-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: wordpress-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": wordpress
+            app.kubernetes.io/name: wordpress
+      toPorts:
+        - ports:
+            - port: "80"
+              protocol: TCP

kubernetes/apps/gatus/configmap.yaml

@@ -160,6 +160,24 @@ data:
         conditions:
           - "[STATUS] == any(200, 301, 302)"
 
+      # Terminus admin UI. Checks the TLS route only — the cleartext
+      # device endpoint (trmnl-device.renner.uno) is the same pod, so a
+      # failure here covers both. Terminus exposes no health endpoint, so
+      # this is a root-page check: the login page answers 200, and an
+      # unauthenticated request may redirect.
+      - name: Web Service 10
+        group: services
+        url: https://trmnl.renner.uno
+        interval: 60s
+        ui:
+          hide-hostname: true
+          hide-url: true
+          hide-port: true
+          hide-conditions: true
+          hide-errors: true
+        conditions:
+          - "[STATUS] == any(200, 301, 302)"
+
       # Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
       # InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
       # no longer deployed. Re-add when/if they come back.

kubernetes/apps/home-assistant/deployment.yaml

@@ -38,7 +38,7 @@ spec:
       # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.1 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/ical-proxy/deployment.yaml

@@ -17,7 +17,33 @@ spec:
     spec:
       containers:
         - name: ical-proxy
-          image: ghcr.io/konairius/ical-proxy:latest
+          # Pinned to a commit build rather than :latest.
+          #
+          # :latest never rolls. The tag moves when a build publishes, but
+          # nothing tells the Deployment about it: the pod template is
+          # unchanged, so Kubernetes has no reason to replace anything, and
+          # this image is deliberately NOT enrolled in Flux image automation
+          # — see kubernetes/infrastructure/image-automation/README.md, which
+          # lists ghcr.io/konairius/ical-proxy as having "only main, latest,
+          # and main-<sha> commit builds; no release tags", so there is no
+          # ImagePolicy watching it. The pods therefore sat on a 21-day-old
+          # image while :latest moved underneath them, and the only way to
+          # pick up a new build was a manual restart.
+          #
+          # A pinned tag makes the rollout a git change: Flux applies the new
+          # pod template and Kubernetes rolls it, which is also what makes
+          # what is RUNNING recoverable from this repo rather than "whatever
+          # :latest meant the last time a pod happened to restart".
+          #
+          # main-99a912b is the merge of konairius/ical-proxy#12, which adds
+          # the summary_match/summary_replace parameters this deployment
+          # needs to strip "| Abfuhrkalender - Landkreis Amberg-Sulzbach"
+          # from every waste collection.
+          #
+          # Bumping this is a manual, reviewed edit. The tag format is
+          # main-<short-sha> (docker/metadata-action type=sha with the branch
+          # as prefix).
+          image: ghcr.io/konairius/ical-proxy:main-99a912b
           ports:
             - containerPort: 8080
               name: http

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -73,3 +73,29 @@ spec:
       interval: 30s
   podTargetLabels:
     - role
+---
+# wp-pg, restored into the `woodpecker` namespace (issue #433). It has never
+# been scraped: `apps` had no PodMonitor for it AND its baseline network policy
+# never allowed the monitoring namespace, so `cnpg_collector_up` has series for
+# the four clusters above but not this one. The namespace move fixes the policy
+# half; without this PodMonitor there is still no scrape target at all.
+#
+# Deliberately not adding one for the cluster still in `apps`: it is retired in
+# stage 3, and a PodMonitor there would only be reconciled to be deleted again.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-wp-pg
+  namespace: woodpecker
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -4,6 +4,8 @@ kind: Kustomization
 # come from the kube-prometheus-stack chart. Flux applies this Kustomization
 # only after `monitoring` is Ready (chart installed → CRDs present).
 resources:
+  - prometheusrule-ceph.yaml
+  - prometheusrule-cilium.yaml
   - prometheusrule-hetzner-backup.yaml
   - prometheusrule-loki.yaml
   - prometheusrule-netbird-availability.yaml

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -0,0 +1,56 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: ceph
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: ceph.rules
+      rules:
+        - alert: CephHealthWarn
+          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
+          # 6h is deliberately long. Ceph warns for self-clearing conditions —
+          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
+          # below its disk threshold — and paging on those trains people to
+          # ignore the alert. What matters is a warning that does not clear.
+          expr: ceph_health_status == 1
+          for: 6h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Ceph has been HEALTH_WARN for 6h"
+            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+
+        - alert: CephHealthError
+          # HEALTH_ERR means data availability or redundancy is at risk.
+          expr: ceph_health_status == 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Ceph is HEALTH_ERR"
+            description: "The Ceph cluster is in HEALTH_ERR — availability or redundancy is at risk. Inspect immediately with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and `ceph osd df tree`."
+
+        - alert: CephMonDiskLow
+          # Named explicitly rather than left to CephHealthWarn, because the
+          # mons sit on the node's /var alongside container storage: this
+          # fires for a full node, not for Ceph growth. The mon's own data is
+          # ~106 MB, so 30% of a 60 GB shared filesystem says nothing about
+          # Ceph.
+          # Image GC does not clear this — measured on k8s-2, the reclaimable
+          # content store is a small fraction of /var/lib/containerd next to
+          # ~20 GB of overlayfs snapshots held by running containers. Look at
+          # what actually grew before reaching for a threshold.
+          expr: |
+            100 * node_filesystem_avail_bytes{mountpoint="/var"}
+              / node_filesystem_size_bytes{mountpoint="/var"} < 30
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.instance }} /var below 30% free — Ceph mon will warn"
+            description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."

kubernetes/apps/monitoring-rules/prometheusrule-cilium.yaml

@@ -0,0 +1,44 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cilium
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cilium.rules
+      # Both alerts are map-agnostic on purpose. cilium_bpf_map_pressure is
+      # emitted per map type — policy, conntrack, NAT, LB — distinguished by
+      # the map_name label, and running out of any of them degrades the
+      # datapath. The policy map is merely the one that has bitten us.
+      # Cilium only exports the metric for maps above
+      # policyMapPressureMetricsThreshold (chart default 0.1), so quiet maps
+      # produce no series rather than a flood of near-zero ones.
+      rules:
+        - alert: CiliumBPFMapPressureHigh
+          # cilium_bpf_map_pressure is used/capacity for the fullest map of
+          # that name on the agent. A full map is not a slow path — it is a
+          # hard failure: the agent cannot program the datapath, so endpoint
+          # regeneration fails and every new pod on the node hangs in sandbox
+          # creation until something frees space.
+          expr: cilium_bpf_map_pressure > 0.8
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Cilium BPF map {{ $labels.map_name }} is {{ $value | humanizePercentage }} full on {{ $labels.node }}"
+            description: "The BPF map {{ $labels.map_name }} on node {{ $labels.node }} has been over 80% full for 15 minutes. At 100% the Cilium agent can no longer program the datapath and new pods on this node fail sandbox creation. Raise the corresponding size in kubernetes/infrastructure/cilium/helmrelease.yaml (values.bpf) — the map is sized at creation, so the change needs an agent restart."
+
+        - alert: CiliumBPFMapFull
+          # Pressure reaches (and can exceed) 1.0 when the agent wants more
+          # entries than the map holds. Pods are already failing to start.
+          expr: cilium_bpf_map_pressure >= 1
+          for: 0m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Cilium BPF map {{ $labels.map_name }} is full on {{ $labels.node }}"
+            description: "The BPF map {{ $labels.map_name }} on node {{ $labels.node }} is at or over capacity. Endpoint regeneration is failing with 'no space left on device' and pods scheduled to this node will hang in ContainerCreating. Check `{namespace=\"kube-system\",pod=~\"cilium-.+\"} |= \"no space left on device\"` in Loki for the affected endpoints."

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/apps/music-assistant/deployment.yaml

@@ -52,7 +52,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -62,7 +62,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/apps/paperless/gotenberg.yaml

@@ -22,7 +22,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: gotenberg
-          image: gotenberg/gotenberg:8.35.0 # {"$imagepolicy": "flux-system:gotenberg"}
+          image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
           args:
             - "gotenberg"
             - "--chromium-disable-javascript=true"

kubernetes/apps/trmnl/cluster.yaml

@@ -0,0 +1,41 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: tr-pg
+  namespace: trmnl
+spec:
+  instances: 1
+  smartShutdownTimeout: 10
+  stopDelay: 60
+  # PG 18, NOT the repo-standard 16. Terminus ships a committed
+  # config/db/structure.sql containing `SET transaction_timeout = 0;` — a GUC
+  # that does not exist before PostgreSQL 17, so schema load against a 16
+  # server dies with `unrecognized configuration parameter`. Upstream's
+  # compose.yml develops against 18.4, so 18 is the tested target; 17 is the
+  # bare minimum. Bare major tag matches the other clusters (minor security
+  # bumps flow automatically, major upgrades stay an explicit decision).
+  imageName: ghcr.io/cloudnative-pg/postgresql:18
+  enableSuperuserAccess: true
+  storage:
+    storageClass: ceph-block
+    size: 5Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: tr-pg-store
+  bootstrap:
+    initdb:
+      database: terminus
+      owner: terminus
+  postgresql:
+    parameters:
+      shared_buffers: "128MB"
+      max_connections: "50"
+      wal_compression: "on"
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      memory: 512Mi

kubernetes/apps/trmnl/device-service.yaml

@@ -0,0 +1,48 @@
+# Device-facing endpoint. Cleartext HTTP, deliberately.
+#
+# The TRMNL firmware initiates every request and cannot be handed the
+# internal CA, so terminating TLS here would simply break polling. Scope of
+# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
+# record), it carries no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
+# UI stays on the TLS route at trmnl.renner.uno.
+#
+# Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
+# pool rather than a TLS-less Ingress, which would be a new pattern for this
+# repo. The device is provisioned with the HOSTNAME, not the address.
+#
+# The address is requested explicitly, matching every other LB service here.
+# Left to allocate freely, Cilium IPAM handed out the first free address in
+# the pool block (10.2.99.0-10.2.99.255) — which is 10.2.99.0, the base
+# address of the range. Each LB address is advertised to the UDM as its own
+# BGP route so that is routable in principle, but a .0 host address is the
+# kind of thing constrained embedded IP stacks get wrong, and the TRMNL
+# firmware is the single client that has to work here. Not worth the risk
+# for an address nobody chose on purpose.
+#
+# .89 continues the existing sequence (.81-.88 are in use) and matches the
+# fd99::89 that IPAM had already picked for v6.
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus-device
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+    lb-pool: bgp
+  annotations:
+    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
+    lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: terminus
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/trmnl/externalsecret.yaml

@@ -0,0 +1,18 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: terminus-secrets
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: terminus-secrets
+  # Holds APP_SECRET only (session/cookie/CSRF key). DATABASE_URL is not in
+  # here — it comes straight off the CNPG-generated `tr-pg-app` Secret's
+  # `uri` key, so the password rotates with the cluster and is never copied.
+  dataFrom:
+    - extract:
+        key: apps/trmnl

kubernetes/apps/trmnl/hetzner-creds-externalsecret.yaml

@@ -0,0 +1,15 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: seed/trmnl/hetzner-creds

kubernetes/apps/trmnl/ingress.yaml

@@ -0,0 +1,28 @@
+# Admin UI only — login, device registration, playlists, screen previews.
+# The device never touches this route: the firmware speaks plain HTTP and
+# would reject the internal CA, so it uses the cleartext LoadBalancer in
+# device-service.yaml instead. Both front the same Service.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: terminus
+  namespace: trmnl
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - trmnl.renner.uno
+      secretName: trmnl-tls
+  rules:
+    - host: trmnl.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: terminus
+                port:
+                  number: 80

kubernetes/apps/trmnl/kustomization.yaml

@@ -0,0 +1,20 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - pvcs.yaml
+  - valkey.yaml
+  - terminus.yaml
+  - ingress.yaml
+  - device-service.yaml
+  # Headless HA screenshotter. Lives in this namespace because Terminus
+  # FETCHES rendered screens back from it (uri delivery mode), so the
+  # two need the intra-namespace path.
+  - renderer.yaml
+  - renderer-externalsecret.yaml
+  # tr-pg CNPG Postgres + offsite Barman backup. Note this cluster runs
+  # PostgreSQL 18, not the repo-standard 16 — see the comment in cluster.yaml.
+  - cluster.yaml
+  - objectstore.yaml
+  - scheduled-backup.yaml
+  - hetzner-creds-externalsecret.yaml

kubernetes/apps/trmnl/objectstore.yaml

@@ -0,0 +1,30 @@
+---
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: tr-pg-store
+  namespace: trmnl
+spec:
+  configuration:
+    # Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
+    # clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
+    # (plugin #646), so at-rest encryption is unavailable; transit is TLS.
+    destinationPath: s3://renner-uno-k8s-cnpg/tr-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      # Region is required (REGION=fsn1) — without it boto3's HeadBucket
+      # (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  retentionPolicy: "30d"

kubernetes/apps/trmnl/pvcs.yaml

@@ -0,0 +1,57 @@
+---
+# Uploads + synchronised fonts for the Terminus pod. Mounted twice by
+# subPath (uploads/, fonts/) into both the web and sidekiq containers, which
+# share one pod — so RWO is sufficient and the Deployment uses `Recreate`.
+#
+# Contents: rendered device screens and the font set pulled by the
+# FONT_SYNCHRONIZER job. Persisting fonts here (rather than an emptyDir)
+# means a pod restart does not depend on upstream being reachable.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: terminus-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 5Gi
+---
+# Valkey RDB snapshots. The Sidekiq queue is reconstructable — the
+# synchroniser jobs re-enqueue on boot — so this is small and losing it is
+# an inconvenience, not data loss.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: terminus-valkey-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 1Gi
+---
+# Renderer state: schedules.json plus the rendered screens under
+# output/, which Terminus fetches back over HTTP in uri delivery mode.
+#
+# This mount is REQUIRED, not just nice to have. The app picks its data
+# directory by testing whether /data exists — with nothing mounted it
+# silently falls back to /app/data inside the container, so schedules
+# and rendered output would vanish on every restart and the panel would
+# go blank until the next render.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: trmnl-renderer-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 2Gi

kubernetes/apps/trmnl/renderer-externalsecret.yaml

@@ -0,0 +1,27 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: trmnl-renderer-secrets
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: trmnl-renderer-secrets
+  # Kept separate from apps/trmnl (which holds Terminus's APP_SECRET)
+  # because these are credentials the RENDERER holds about other
+  # systems, with different rotation stories:
+  #
+  #   HA_ACCESS_TOKEN     — a dedicated Home Assistant long-lived token,
+  #                         not the operator's personal one, so the
+  #                         renderer can be revoked in isolation.
+  #   BYOS_ACCESS_TOKEN   — Terminus JWTs, minted once out of band from
+  #   BYOS_REFRESH_TOKEN    a Terminus login. The PASSWORD deliberately
+  #                         never enters the cluster; stock Terminus
+  #                         disables session expiry, so these are
+  #                         effectively permanent.
+  dataFrom:
+    - extract:
+        key: apps/trmnl/renderer

kubernetes/apps/trmnl/renderer.yaml

@@ -0,0 +1,385 @@
+# Headless renderer: screenshots the Home Assistant e-ink dashboard,
+# dithers it to 16 greys and hands it to Terminus.
+#
+# Upstream ships this as a Home Assistant add-on. Our HA is a plain
+# container with no Supervisor, so it runs standalone here instead —
+# viable because the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env
+# vars over /data/options.json and authenticates with an ordinary
+# long-lived token. SUPERVISOR_TOKEN is deliberately NOT set: it only
+# flips an internal isAddOn flag, and setting it would switch the app
+# to add-on behaviour (fixed port, /data assumptions).
+#
+# DIRECTION OF TRAVEL, which is the non-obvious part: with the
+# byos-hanami webhook in `uri` delivery mode the renderer does NOT
+# push image bytes. It POSTs a URL pointing at its own /output/<file>
+# endpoint, and TERMINUS FETCHES THE IMAGE BACK from it. That is why
+# both live in this namespace — the pull is intra-namespace and is
+# already permitted by the default-deny-ingress rule.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: trmnl-renderer-schedule
+  namespace: trmnl
+data:
+  # Template for /data/schedules.json. The init container splices the
+  # BYOS tokens in; everything non-secret is declared here so the
+  # schedule is reviewable in git rather than clicked into the app's
+  # web UI (which has no authentication and is not exposed).
+  #
+  # The app re-reads schedules.json every 60 s, so edits here take
+  # effect without a restart once the init container has rewritten it.
+  #
+  # lang is "de", not null. It calls Home Assistant's own
+  # _selectLanguage() for the screenshot session, so the rendered page
+  # is localised by HA rather than by hand. With it left null the
+  # dashboard grew a German weekday array, a German month array and a
+  # fifteen-entry weather-condition dict — reimplementing translations
+  # HA already ships. The instance itself stays language "en"; this
+  # only affects the render session.
+  #
+  # model_id is 1 — TRMNL X (name "v2", 1872x1404, 16 colours) — and it
+  # is an INTEGER, not a name. Confirmed against the running server:
+  # GET /api/screens returns model_id as a number, and upstream's
+  # example value "og_1bit" matches no model on this instance (the
+  # 2-bit OG model here is id 11, name "og_plus"). Getting this wrong
+  # shows up as Terminus rejecting the pushed screen with a 422.
+  #
+  # All three auth fields below are placeholders the init container
+  # replaces — including obtained_at, which must end up a real epoch-MS
+  # stamp. The 0 here is inert only because it is always overwritten;
+  # left as 0 the scheduler reads the tokens as ancient and refuses to
+  # push at all ("expired beyond the refresh window"), with no render
+  # attempt and no error beyond one startup warning.
+  schedules.template.json: |
+    [
+      {
+        "id": "trmnl-entrance",
+        "name": "Entrance e-ink panel",
+        "enabled": true,
+        "cron": "* * * * *",
+        "interval_minutes": 1,
+        "ha_mode": true,
+        "dashboard_path": "/trmnl-entrance/home",
+        "target_url": null,
+        "viewport": { "width": 1872, "height": 1404 },
+        "device": "schedule_trmnl_x",
+        "crop": { "enabled": false, "x": 0, "y": 0, "width": 0, "height": 0 },
+        "format": "png",
+        "rotate": null,
+        "zoom": 1,
+        "wait": null,
+        "theme": "Graphite E-ink Light",
+        "lang": "de",
+        "dark": false,
+        "invert": false,
+        "timestamp": false,
+        "dithering": {
+          "enabled": true,
+          "method": "floyd-steinberg",
+          "palette": "gray-16",
+          "bitDepth": 4,
+          "gammaCorrection": true,
+          "blackLevel": 0,
+          "whiteLevel": 100,
+          "normalize": true,
+          "saturationBoost": false
+        },
+        "webhook_url": "http://terminus.trmnl.svc.cluster.local/api/screens",
+        "webhook_headers": {},
+        "webhook_format": {
+          "format": "byos-hanami",
+          "byosConfig": {
+            "label": "Eingang",
+            "name": "trmnl-entrance",
+            "model_id": 1,
+            "preprocessed": true,
+            "addon_base_url": "http://trmnl-renderer.trmnl.svc.cluster.local:10000",
+            "delivery_mode": "uri",
+            "auth": {
+              "enabled": true,
+              "access_token": "__BYOS_ACCESS_TOKEN__",
+              "refresh_token": "__BYOS_REFRESH_TOKEN__",
+              "obtained_at": 0
+            }
+          }
+        },
+        "createdAt": "2026-08-11T00:00:00.000Z",
+        "updatedAt": "2026-08-11T00:00:00.000Z"
+      }
+    ]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: trmnl-renderer
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: trmnl-renderer
+  annotations:
+    # BOTH sources need this, and for the same reason: the init
+    # container writes /data/schedules.json ONCE, at pod start. Nothing
+    # re-reads either source afterwards.
+    #
+    # The app re-reads schedules.json every 60s, which makes it look
+    # self-updating — but only the FILE, and the file is only rewritten
+    # by the init container. So a ConfigMap or Secret change lands in
+    # git, updates the object, and the running renderer keeps the old
+    # values indefinitely.
+    #
+    # Both halves were hit repeatedly in practice and each time needed
+    # `kubectl delete pod` by hand: the theme name, the schedule's lang
+    # field, and every re-mint of the BYOS tokens. A fix that depends on
+    # someone remembering to restart a pod is not deployed.
+    #
+    # Named per-resource rather than the blanket
+    # reloader.stakater.com/auto: "true", matching terminus.yaml and
+    # mosquitto-deployment.yaml.
+    configmap.reloader.stakater.com/reload: trmnl-renderer-schedule
+    secret.reloader.stakater.com/reload: trmnl-renderer-secrets
+spec:
+  replicas: 1
+  # RWO PVC, and two renderers would push conflicting screens for the
+  # same device.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: trmnl-renderer
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: trmnl-renderer
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        # Writes /data/schedules.json from the ConfigMap template with
+        # the BYOS tokens injected.
+        #
+        # jq rather than sed: the tokens are JWTs and go into a JSON
+        # string, so they need JSON-safe escaping. A sed substitution
+        # would corrupt on any character sed treats as special.
+        #
+        # No HTTP here on purpose. The app does expose a /api/byos/login
+        # proxy that exchanges a Terminus username+password for these
+        # tokens, but that would mean shipping the Terminus PASSWORD
+        # into the cluster and doing a network call at every pod start.
+        # Tokens are therefore minted once out of band and kept in
+        # OpenBao — the password never enters the cluster.
+        #
+        # That only works because Terminus is configured with
+        # API_ACCESS_TOKEN_PERIOD = 10 years (see terminus.yaml). It is
+        # NOT, as an earlier version of this comment claimed, because
+        # stock Terminus issues permanent tokens — the stock default is
+        # 1800s, and this deployment demonstrated it by having the
+        # webhook start failing with "expired JWT access token" about
+        # thirty minutes after seeding.
+        - name: render-schedule
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              # obtained_at is stamped at write time, in MILLISECONDS.
+              # It is not cosmetic: the scheduler derives token age from
+              # it, and a literal 0 made the app treat freshly-minted
+              # tokens as ancient and skip the webhook entirely —
+              #   "BYOS tokens ... expired beyond the refresh window"
+              #
+              # Stamping it "now" is only safe because Terminus is
+              # configured with API_ACCESS_TOKEN_PERIOD = 10 years (see
+              # terminus.yaml). Against the 30-minute default it would be
+              # actively harmful: the app would believe a stale token had
+              # 30 fresh minutes left, skip its proactive refresh, and
+              # then be unable to recover, because refreshing requires a
+              # still-valid token. If that setting is ever reverted, this
+              # must become a real mint time carried alongside the tokens.
+              jq --arg at "$BYOS_ACCESS_TOKEN" \
+                 --arg rt "$BYOS_REFRESH_TOKEN" \
+                 '(.[].webhook_format.byosConfig.auth.access_token) = $at
+                  | (.[].webhook_format.byosConfig.auth.refresh_token) = $rt
+                  | (.[].webhook_format.byosConfig.auth.obtained_at) = (now * 1000 | floor)' \
+                 /template/schedules.template.json > /data/schedules.json
+              echo "wrote /data/schedules.json ($(wc -c < /data/schedules.json) bytes)"
+          env:
+            - name: BYOS_ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trmnl-renderer-secrets
+                  key: BYOS_ACCESS_TOKEN
+            - name: BYOS_REFRESH_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trmnl-renderer-secrets
+                  key: BYOS_REFRESH_TOKEN
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            - name: schedule-template
+              mountPath: /template
+              readOnly: true
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      containers:
+        # Pinned, not enrolled in image automation — same reasoning as
+        # Terminus itself. This drives a wall-mounted panel and upstream
+        # is pre-1.0; an unattended bump that changes the dithering
+        # pipeline or the webhook payload shows up as a corrupted
+        # screen, not as a failed rollout.
+        #
+        # Note the image is ARCH-SUFFIXED: upstream publishes no
+        # multi-arch manifest, only trmnl-ha-amd64 and trmnl-ha-aarch64.
+        - name: renderer
+          image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.9.5
+          ports:
+            - name: http
+              containerPort: 10000
+          env:
+            # The in-cluster Service name, NOT ha.renner.uno and not
+            # home.renner.uno. Both public names are unusable from a
+            # pod: ha.renner.uno resolves to the home-assistant-bgp
+            # LoadBalancer's external address (10.2.99.87), which is not
+            # backed by an Ingress, so CoreDNS's k8s_gateway delegation
+            # does not intercept it and the pod hairpins off the LB.
+            #
+            # This cluster has hit that exact failure before — see the
+            # comment on kubernetes/infrastructure/openbao/rotation/
+            # home-assistant.yaml, which reaches HA the same way and
+            # states it outright: "The public name is unusable from a
+            # pod (ingress-LB hairpin)."
+            #
+            # FAILS QUIETLY if wrong: the schedule keeps firing every
+            # minute, each render errors on connect, and the panel just
+            # never updates. Nothing goes CrashLoopBackOff.
+            - name: HOME_ASSISTANT_URL
+              value: "http://home-assistant-bgp.home-assistant.svc.cluster.local:8123"
+            - name: TZ
+              value: "Europe/Berlin"
+            # Chromium will not start without a writable HOME. Running
+            # as uid 1000 with no matching passwd entry leaves HOME="/",
+            # which is root-owned 0755, and the browser dies on launch
+            # with
+            #   chrome_crashpad_handler: --database is required
+            # because crashpad cannot create its database directory.
+            #
+            # Diagnosed on the running pod, not guessed: HOME=/, uid=1000,
+            # / not writable, /tmp writable. Crash dumps are throwaway, so
+            # /tmp is the right home for them rather than the PVC.
+            #
+            # This is a consequence of OUR non-root securityContext, not
+            # of the image — so it must move together with runAsUser.
+            - name: HOME
+              value: "/tmp"
+            # Dedicated long-lived token, NOT the operator's personal
+            # one, so the renderer's access can be revoked on its own.
+            # It is injected into the headless browser's localStorage,
+            # which is how the app authenticates to the HA frontend.
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trmnl-renderer-secrets
+                  key: HA_ACCESS_TOKEN
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            # Bounded scratch backing HOME. Chromium writes crashpad
+            # state and profile data here on every launch, and it now
+            # launches once a minute for the life of the pod — on the
+            # container's writable layer that is unbounded growth
+            # against node disk, with no ephemeral-storage limit to
+            # catch it. A sizeLimit converts a slow, cluster-wide
+            # node-disk-pressure problem into a bounded and clearly
+            # attributable eviction of this pod.
+            - name: tmp
+              mountPath: /tmp
+          # HTTP probes here, unlike terminus.yaml's TCP-only ones, because
+          # /health is confirmed rather than assumed: the upstream image's
+          # own Dockerfile declares
+          #   HEALTHCHECK ... curl -f http://localhost:10000/health
+          # so the route is part of the image's published contract. Terminus
+          # got TCP probes precisely because it publishes no such endpoint.
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 250m
+              memory: 768Mi
+            # Chromium is a SEPARATE process from the app, and the app's
+            # own watchdog only measures its own RSS (restarting itself
+            # past 900 MB). The limit therefore has to cover the app's
+            # ceiling PLUS a browser rendering a 1872x1404 page, which
+            # is well above the 512 MB upstream's troubleshooting note
+            # suggests.
+            limits:
+              memory: 2Gi
+          # Chromium normally needs either SYS_ADMIN or a user namespace to
+          # start its own sandbox, which would make dropping ALL fatal. It
+          # is safe here because the app launches the browser with
+          # --no-sandbox --disable-setuid-sandbox --no-zygote (and
+          # --disable-dev-shm-usage, which is why no /dev/shm emptyDir is
+          # mounted either). Those flags are baked into the app's own
+          # PUPPETEER_ARGS, not something we pass, so this holds as long as
+          # the pinned tag does — worth re-checking on any bump.
+          #
+          # Confirmed on the running pod: the capability drop is genuinely
+          # not what broke the browser. The first deploy DID crash it, but
+          # on a writable-HOME problem caused by runAsUser — see the HOME
+          # env above. Do not "fix" a future Chromium launch failure by
+          # loosening this block before checking HOME and /tmp.
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: trmnl-renderer-data
+        - name: schedule-template
+          configMap:
+            name: trmnl-renderer-schedule
+        - name: tmp
+          emptyDir:
+            sizeLimit: 512Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: trmnl-renderer
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: trmnl-renderer
+spec:
+  # ClusterIP only, and deliberately no Ingress. The app's web UI on
+  # this port has NO authentication of any kind — upstream says so
+  # outright — so it must not be routable from the LAN. Terminus
+  # reaches it in-namespace to fetch rendered screens; nothing else
+  # needs to.
+  selector:
+    app.kubernetes.io/name: trmnl-renderer
+  ports:
+    - name: http
+      port: 10000
+      targetPort: http

kubernetes/apps/trmnl/scheduled-backup.yaml

@@ -0,0 +1,16 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: tr-pg-daily
+  namespace: trmnl
+spec:
+  # 03:00 local, same slot as the other CNPG clusters (the Velero PVC
+  # schedule for this namespace runs later, at 08:15).
+  schedule: "0 0 3 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: tr-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/trmnl/terminus.yaml

@@ -0,0 +1,256 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: terminus-config
+  namespace: trmnl
+data:
+  TZ: "Europe/Berlin"
+  HANAMI_PORT: "2345"
+  # What the device is told to fetch from. Terminus bakes this into the
+  # absolute `image_url` it hands back on /api/display, so it MUST be the
+  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
+  # HTTP and would reject the internal CA. No trailing slash: the same
+  # string is typed into the captive portal at provisioning time.
+  API_URI: "http://trmnl-device.renner.uno"
+  KEYVALUE_URL: "redis://terminus-valkey:6379"
+  # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
+  # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB
+  # pool (fd99::/120) and the LAN ULA — without it every IPv6 client,
+  # including the device and Traefik, is rate-limited as if it were remote.
+  RACK_ATTACK_ALLOWED_SUBNETS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1,::1,fc00::/7"
+  # 10 years, against a default of 1800s (30 minutes).
+  #
+  # Documented upstream in doc/configuration.adoc:
+  #   API_ACCESS_TOKEN_PERIOD — "The amount of time, in seconds, the JWT
+  #   access token remains valid before needing to refresh your access
+  #   token." Default: 1,800 seconds (30 minutes).
+  # Corroborated on this instance: a token minted here was rejected about
+  # thirty minutes later, i.e. the documented default behaving as written.
+  #
+  # This exists for the renderer, which pushes screens to /api/screens with
+  # a JWT held in OpenBao. At the default period that arrangement cannot
+  # work unattended, and it fails in a way that has no recovery: minting a
+  # replacement token via POST /api/jwt requires a still-VALID access token
+  # in the Authorization header, so once one expires it cannot be refreshed
+  # from itself. Observed exactly that on first deploy —
+  #   [webhook] Webhook failed: 400 {"error":"expired JWT access token"}
+  #   [webhook] BYOS auth: refresh failed (400) {"error":"expired JWT ..."}
+  # — the refresh attempt being rejected by the very expiry it was trying
+  # to repair.
+  #
+  # The renderer's own code assumes "stock Terminus disables session
+  # expiration and issues effectively permanent tokens". That is simply not
+  # true of this deployment, so the assumption is made true here rather
+  # than worked around in the client.
+  #
+  # The trade is deliberate: a long-lived JWT with full Terminus API rights
+  # now sits in OpenBao. Accepted because the alternative was putting the
+  # Terminus account PASSWORD into the cluster so an init container could
+  # mint a short-lived token on every pod start, which is a strictly worse
+  # secret to hold. The token is revocable by rotating the account's
+  # credentials; it grants access only to this LAN-only instance.
+  #
+  # NOTE: the period is fixed at mint time. Changing this does not extend
+  # tokens that already exist — they must be re-minted afterwards.
+  API_ACCESS_TOKEN_PERIOD: "315360000"
+  # The OTHER half of the same problem, and the reason the first fix
+  # did not hold. Terminus has TWO independent expiries: the JWT's own
+  # lifetime (above) and the SESSION behind it. Raising only the token
+  # period left the session at its defaults, so pushes kept working for
+  # a while and then failed with
+  #   HTTP 401 {"error":"This session has expired, please login again"}
+  # which is a different error, from a different mechanism, than the
+  # "expired JWT access token" the first fix addressed.
+  #
+  # Documented in doc/configuration.adoc:
+  #   SESSION_EXPIRATION_ENABLED — "Determines if your browser session
+  #   or JWT API token is expired or not." Default: true.
+  #   SESSION_INACTIVITY_LIMIT 1800s, SESSION_LIFETIME_LIMIT 86400s.
+  #
+  # Disabling it is exactly the state the renderer's own code assumes
+  # ("stock Terminus disables session expiration and issues ~100-year
+  # tokens") — an assumption that is false by default and has to be
+  # made true here, same as the token period.
+  #
+  # Same trade as above, and it is the deciding one for an unattended
+  # appliance: a session that expires needs a human with a password to
+  # restore it, and this panel has neither.
+  SESSION_EXPIRATION_ENABLED: "false"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: terminus
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+  annotations:
+    # terminus-config is consumed via envFrom, and env vars are read ONCE at
+    # container start — Kubernetes does not hot-reload them. The ConfigMap is
+    # a plain resource, not a configMapGenerator, so editing it does not
+    # change the pod template hash and Flux will not roll the pods either.
+    # Without this annotation a ConfigMap-only change (such as raising
+    # API_ACCESS_TOKEN_PERIOD) applies to git and to the ConfigMap object
+    # while the running app keeps the old value indefinitely — a silent
+    # no-op, with the manifest and reality disagreeing.
+    #
+    # Named per-ConfigMap rather than the blanket `reloader.stakater.com/
+    # auto: "true"`, matching mosquitto-deployment.yaml: enrolling anything
+    # further stays a visible, reviewable edit to this line.
+    configmap.reloader.stakater.com/reload: terminus-config
+spec:
+  replicas: 1
+  # RWO PVC shared by both containers in this pod, and APP_SETUP runs
+  # migrations on boot — two concurrent web pods would race on both.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: terminus
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: terminus
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        # Tag pinned deliberately and NOT enrolled in Flux image automation.
+        # Terminus is pre-1.0 with no GitHub Releases and no changelog; an
+        # unattended bump that changes the firmware or /api/display contract
+        # is a device-bricking class of risk. Bumps are a reviewed, manual
+        # act — see kubernetes/infrastructure/image-automation/README.md.
+        - name: terminus
+          image: ghcr.io/usetrmnl/terminus:0.68.0
+          ports:
+            - name: http
+              containerPort: 2345
+          envFrom:
+            - configMapRef:
+                name: terminus-config
+          env:
+            # Runs `hanami assets compile` + `hanami db migrate` in the
+            # entrypoint before exec'ing puma. Set on the web container
+            # only — the sidekiq sidecar must not race the migration.
+            - name: APP_SETUP
+              value: "true"
+            # CNPG generates this Secret; `uri` already carries the rotated
+            # password and points at the tr-pg-rw service.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: tr-pg-app
+                  key: uri
+            - name: APP_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: terminus-secrets
+                  key: APP_SECRET
+          volumeMounts:
+            - name: data
+              mountPath: /app/public/uploads
+              subPath: uploads
+            - name: data
+              mountPath: /app/public/fonts
+              subPath: fonts
+          # TCP probes, not HTTP: Terminus exposes no documented health
+          # endpoint, and every real route is behind either auth or
+          # Rack::Attack. Tighten to an HTTP GET once the route set is
+          # confirmed against the running app.
+          startupProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 10
+            # ~10 min: first boot compiles assets and migrates a fresh DB.
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              memory: 1Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+        - name: sidekiq
+          image: ghcr.io/usetrmnl/terminus:0.68.0
+          command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
+          envFrom:
+            - configMapRef:
+                name: terminus-config
+          # EXPECTED on a fresh deploy: this container loses the race against
+          # the web container's migration and exits 1 with
+          # `PG::UndefinedTable: relation "account" does not exist`, then
+          # comes up healthy on the restart once the schema exists.
+          # Observed on first deploy 2026-08-10; converged in ~1s. There is
+          # no upstream ordering hook, and gating it behind an initContainer
+          # that probes for a specific table would couple us to Terminus's
+          # schema. So this is left to Kubernetes' restart, deliberately —
+          # a single sidekiq restart shortly after a rollout is benign. More
+          # than a couple, or restarts long after a rollout, are NOT this.
+          env:
+            # No APP_SETUP here — see the web container.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: tr-pg-app
+                  key: uri
+            - name: APP_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: terminus-secrets
+                  key: APP_SECRET
+          volumeMounts:
+            - name: data
+              mountPath: /app/public/uploads
+              subPath: uploads
+            - name: data
+              mountPath: /app/public/fonts
+              subPath: fonts
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 512Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: terminus-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+spec:
+  selector:
+    app.kubernetes.io/name: terminus
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/trmnl/valkey.yaml

@@ -0,0 +1,79 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: terminus-valkey
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus-valkey
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: terminus-valkey
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: terminus-valkey
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 999
+        fsGroup: 999
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        # Valkey major version tracks upstream's compose.yml (9.x). Sidekiq
+        # is the only consumer; a major bump is a separate migration.
+        - name: valkey
+          image: valkey/valkey:9.1.1-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
+          args:
+            - valkey-server
+            - --save
+            - "60"
+            - "1"
+            - --loglevel
+            - warning
+          ports:
+            - name: valkey
+              containerPort: 6379
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          livenessProbe:
+            tcpSocket:
+              port: valkey
+            initialDelaySeconds: 10
+          readinessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: terminus-valkey-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus-valkey
+  namespace: trmnl
+spec:
+  selector:
+    app.kubernetes.io/name: terminus-valkey
+  ports:
+    - port: 6379
+      targetPort: valkey

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.5.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -0,0 +1,388 @@
+# Woodpecker: `apps` → `woodpecker` namespace migration
+
+Moves the Woodpecker service — server, agent and the `wp-pg` CNPG backend — out
+of the shared `apps` namespace into one of its own. Distinct from
+`MIGRATION.md`, which covers the earlier SQLite → CNPG move.
+
+## Why
+
+`apps` holds gitea, gatus, external-dns, ical-proxy and woodpecker together,
+and its `default-deny-ingress` uses `endpointSelector: {}`. Every allow rule it
+carries therefore applies to *every* workload in the namespace.
+
+That is not academic. Issue #433 — `wp-pg` stuck at `Ready=False` with
+`Instance Status Extraction Error` for three weeks — is caused by the CNPG
+operator in `database` being unable to reach `https://<pod-ip>:8000/pg/status`.
+The only namespace-level fix is `fromEndpoints: namespace=database`, which
+would expose gitea, gatus, external-dns and ical-proxy to the whole `database`
+namespace to solve a problem affecting one pod on one port.
+
+`cnpg_collector_up` has series for authentik, immich, monitoring and paperless
+but not for `wp-pg`, for the same reason: `apps` never allowed the monitoring
+namespace either.
+
+A namespace per service keeps each grant on the workload that needs it.
+
+**`woodpecker-ci` is not the target.** It enforces `pod-security: privileged`
+and exists solely for ephemeral pipeline step pods running repo-supplied code.
+Putting the database there would let any build step reach it through the
+intra-namespace allow.
+
+## Staging
+
+Rollback is trivial through stage 2. The `apps` cluster is untouched until
+stage 3, so reverting means pointing the DSN back and scaling up.
+
+| Stage | Contents | CI impact |
+|---|---|---|
+| **1** (this PR) | `woodpecker` namespace, its network policies, ExternalSecret, ObjectStores, and `wp-pg` restored from barman | none — parallel, nothing points at it |
+| **runbook** | verify the restore holds real data | none |
+| **2** | move server + agent, flip DSN, RBAC subject, rotator policies, docs | **downtime** |
+| **3** | delete the `apps` originals, add the prune guard to the survivor | none |
+
+## Stage 1 design notes
+
+**Two ObjectStores, deliberately.** `wp-pg-store-source` points at
+`s3://renner-uno-k8s-cnpg/wp-pg` — the prefix the live cluster archives to —
+and is read-only. `wp-pg-store` points at a **new** prefix,
+`s3://renner-uno-k8s-cnpg/wp-pg-woodpecker`, for ongoing archiving.
+
+Both clusters writing one `destinationPath` would interleave their WAL streams
+and corrupt both. Splitting them also leaves the pre-migration backup intact
+and independently restorable.
+
+**`serverName: wp-pg`** on the `externalCluster` names the cluster that
+*produced* the backup, not the one being created.
+
+**The namespace is declared in `infrastructure/namespaces/namespaces.yaml`,**
+not alongside these workloads. Its CiliumNetworkPolicies ship with the
+`infrastructure-crds` KS (`infrastructure/post-crds` includes
+`../network-policies/`), which `dependsOn` `infrastructure` — so the namespace
+exists before any policy targets it. Creating it in the `woodpecker` workload
+KS would invert that order, since that KS runs after both.
+
+**Ingress from `woodpecker-ci` is deliberately absent in stage 1.** Step pods do
+call the server back on `:8000`, but the CNPG instance manager serves its status
+API on `:8000` too, and the baseline policy uses `endpointSelector: {}`. A
+`toPorts` block narrows the port, not the destination workload — so putting that
+rule on the baseline would hand arbitrary repo-supplied build steps a route to
+wp-pg's instance manager. It lands in stage 2 as its own policy selecting the
+server pod.
+
+**No `prune: disabled` guard on the restored cluster.** The original carries one
+because it holds the only copy of the data. This one is reconstructable by
+re-running the restore, and a prune guard on a migration artifact is how orphans
+survive an abandoned migration. It gains the guard in stage 3.
+
+**Barman credentials still come from `seed/paperless/hetzner-creds`.** Issue
+#397 wants woodpecker to own `seed/woodpecker/hetzner-creds`, and this would
+have been the natural moment — but creating that path needs OpenBao write
+access that was not available (`bao kv list seed` → 403). Repointing at a path
+that may not exist would leave the restored cluster unable to archive WAL,
+which is the thing this stage exists to prove. #397 stays open; the swap is one
+line once the path exists.
+
+## Expect the `woodpecker` Kustomization to go not-Ready during the restore
+
+`kubernetes/clusters/homelab/woodpecker.yaml` sets `wait: true` and a
+`healthCheckExprs` entry matching `postgresql.cnpg.io/v1 Cluster` **with no
+name or namespace qualifier**. Flux evaluates it against every Cluster in that
+Kustomization's inventory — which, after this stage, is both the live
+`apps/wp-pg` and the restoring `woodpecker/wp-pg`.
+
+So while the restore runs, the whole `woodpecker` KS — the one that owns
+production Woodpecker — reports not-Ready, and a reconcile timeout in that
+window will trip the `flux-failures` Alert. **That is expected and is not
+Woodpecker breaking.** Nothing `dependsOn: woodpecker`, so it does not cascade.
+
+If the restore is going to sit for a long time, or the noise is unwelcome, the
+options are to suspend the KS while it completes, or to split the migration
+resources into their own `wait: false` Kustomization. For a ~660 MB database the
+window should be minutes.
+
+## Verifying stage 1
+
+Restore is asynchronous. Watch it reach a healthy state:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -w
+kubectl -n woodpecker get pods
+```
+
+Expect `Cluster in healthy state`. If it sticks in bootstrap, the recovery job's
+logs carry the reason:
+
+```sh
+kubectl -n woodpecker logs -l cnpg.io/jobRole=full-recovery --tail=100
+```
+
+**First check the generated app credentials target the right database.** CNPG
+defaults `bootstrap.recovery.database` to `app`; the restored data uses
+`woodpecker`. If this reads anything but `woodpecker`, the cluster has
+provisioned an empty `app` database and the real data is unreachable through
+the `-app` Secret and Service:
+
+```sh
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.dbname}' | base64 -d; echo
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.username}' | base64 -d; echo
+```
+
+**Then confirm the restore actually holds Woodpecker's data** — a cluster that
+comes up healthy but empty is the failure mode worth catching:
+
+```sh
+PGPASS=$(kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c '\dt'
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+Row counts should match the live cluster:
+
+```sh
+PGPASS_OLD=$(kubectl -n apps get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n apps exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS_OLD" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+They will not match exactly if builds ran after the backup the restore used.
+
+**And stage 2 does not close that gap.** An earlier draft of this document
+claimed it quiesced the server first; it does not — Flux simply prunes the
+`apps` workloads and creates the `woodpecker` ones, so anything written to the
+old database between the restore and the cutover stays only in the old database.
+
+Measured on the live cutover: the restore captured through pipeline `9488`,
+four more (`9491`, `9492`, `9494`, `9496`) landed in `apps/wp-pg` before the
+swap, and the new cluster resumed at `9521`. Four pipeline records and their
+log entries exist only in the old cluster. No ID collision, no functional
+breakage — Woodpecker's own manifest calls this database "small and
+non-critical (state is largely reconstructable)" — but it is real history loss
+and should be a decision, not a surprise.
+
+If that gap matters for a future migration, quiesce before cutting over:
+
+```sh
+flux suspend kustomization woodpecker
+kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0
+# re-run the restore so it captures the final state, then resume
+```
+
+The old cluster keeps its `prune: disabled` guard until stage 3, so the missing
+rows remain recoverable until then.
+
+Confirm the two fixes this move exists for:
+
+```sh
+# #433: operator can now reach the instance
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+# metrics now scraped — needs BOTH the PodMonitor (cnpg-podmonitors.yaml) and
+# the monitoring ingress allow; wp-pg had neither in `apps`, which is why it is
+# absent from cnpg_collector_up while ak-pg/im-pg/gf-pg/pprl-pg are present
+kubectl get --raw '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=cnpg_collector_up{namespace="woodpecker"}'
+```
+
+Confirm the scrape target is actually registered, not just that the metric is
+missing for some other reason:
+
+```sh
+kubectl -n woodpecker get podmonitor cnpg-wp-pg
+```
+
+**Check the policy semantics by hand.** `kubectl --dry-run=server` validates
+manifest shape, not Cilium behaviour, so it cannot catch a rule that is
+syntactically fine but scoped wider than intended.
+
+`cilium-dbg policy trace` does **not** exist in Cilium 1.20 — it was removed,
+and `cilium-dbg policy` now offers only `get`, `selectors`, `subject-selectors`
+and `wait`. Read the realized BPF policy map instead, which is better evidence
+anyway: it is the datapath as programmed, not a simulation of it, and it
+carries live byte/packet counters.
+
+Find the agent on wp-pg's node and the endpoint id:
+
+```sh
+kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}'
+kubectl -n kube-system get pod -l k8s-app=cilium \
+  --field-selector spec.nodeName=<node> --no-headers -o custom-columns=N:.metadata.name
+
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg endpoint list -o json \
+  | jq -r '.[] | select((.status.labels."security-relevant"[]?|tostring)
+      | test("cnpg.io/cluster=wp-pg")) | "\(.id) \(.status.networking.addressing[0].ipv4)"'
+```
+
+Then, on that endpoint id — who may reach `:8000`:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | awk '$4 ~ /8000/ || NR<=1'
+```
+
+Expect exactly one row, the CNPG operator, with non-zero counters once it has
+polled:
+
+```
+Allow  Ingress  k8s:app.kubernetes.io/instance=cloudnative-pg  8000/TCP  11756  58
+```
+
+And confirm CI step pods reach **ICMP only** — never a TCP port:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | grep woodpecker-ci | awk '{print $4}' | sort -u
+```
+
+Every line must be an ICMP/ICMPv6 type. A bare port number here means the
+namespace baseline has grown a rule it should not have.
+
+Note the operator's allow is also confirmed indirectly and more strongly by
+`Ready=True` on the Cluster: that condition only goes true once the operator has
+successfully polled this exact endpoint and port.
+
+And that archiving goes to the new prefix, leaving the old one alone:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")]}'
+```
+
+## Stage 2 — the cutover
+
+Moves server, agent, service, ingress, pdb, the agent ServiceAccount, the
+`woodpecker-secret` ExternalSecret and the rotator policy into `woodpecker`,
+points the server's DSN at `wp-pg-rw.woodpecker.svc`, repoints the agent's gRPC
+at `woodpecker.woodpecker.svc:9000`, moves the openbao rotators' egress, and
+adds the `woodpecker-ci` → server `:8000` policy that stage 1 deliberately
+deferred — scoped to the server pod, not the namespace baseline.
+
+**This is the downtime window.** Flux prunes the `apps` copies and creates the
+`woodpecker` ones; `ci.renner.uno` is down in between, and in-flight pipelines
+die with their agent.
+
+**The outage outlasts the pods becoming Ready, because of TLS.** Certificates
+are namespace-scoped, so moving the Ingress does not move `woodpecker-tls` —
+cert-manager issues a fresh one in the new namespace. Until it completes,
+Traefik serves a default certificate and clients fail before they get an HTTP
+status at all:
+
+```
+curl: (60) SSL: no alternative certificate subject name matches target hostname 'ci.renner.uno'
+```
+
+That is expected, not a routing or policy fault. Watch it clear with:
+
+```sh
+kubectl -n woodpecker get certificate woodpecker-tls -w
+```
+
+`letsencrypt-cluster` solves **DNS-01** via Cloudflare, so there is no in-cluster
+ACME solver pod and no HTTP challenge to admit through the network policies —
+worth knowing before scoping any ingress rule, since an HTTP-01 issuer would
+need the solver reachable too.
+
+The four CNPG files (`cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`,
+`hetzner-creds-externalsecret.yaml`) still describe the **original** apps-namespace
+cluster and are deliberately untouched — they are the rollback. Stage 3 retires
+them.
+
+### Verifying stage 2
+
+```sh
+kubectl -n woodpecker get pods
+kubectl -n apps get pods -l app.kubernetes.io/name=woodpecker   # expect none
+```
+
+The server must reach the restored database, not the old one:
+
+```sh
+kubectl -n woodpecker logs deploy/woodpecker-server --tail=40 | grep -i "database\|migrat\|error"
+kubectl -n woodpecker exec deploy/woodpecker-server -- env | grep DATASOURCE
+```
+
+Then confirm the UI and a real pipeline, which is the only check that exercises
+server → agent → step pod → server end to end:
+
+```sh
+curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
+woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+```
+
+Step pods must still reach the server on `:8000` while still being unable to
+reach wp-pg on any TCP port — re-run the `bpf policy get` checks above against
+both the server's and wp-pg's endpoint ids.
+
+Check the rotators can still reach the server, since their egress moved:
+
+```sh
+kubectl -n openbao get cronjob
+kubectl -n openbao logs -l app.kubernetes.io/component=secret-rotator --tail=20
+```
+
+### Rolling back stage 2
+
+Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
+`wp-pg-rw.apps.svc`, which never stopped running. Data written to the new
+cluster during the window is lost, so roll back promptly or not at all — if
+pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
+
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
+## Rolling back stage 1
+
+Delete the four resources and the namespace. Nothing references them, and the
+`apps` cluster is untouched:
+
+```sh
+kubectl delete ns woodpecker
+```
+
+Then revert this PR so Flux stops recreating them.

kubernetes/apps/woodpecker/agent.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -12,7 +12,7 @@ metadata:
     secret.reloader.stakater.com/reload: woodpecker-secret
 spec:
   # 1: the in-cluster server owns ci.renner.uno now, so this agent registers with
-  # it (woodpecker.apps.svc:9000) and runs pipelines on the kubernetes backend,
+  # it (woodpecker.woodpecker.svc:9000) and runs pipelines on the kubernetes backend,
   # replacing the retired docker agents (LXC 910/911).
   replicas: 1
   selector:
@@ -37,7 +37,7 @@ spec:
           image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
-              value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
             - name: WOODPECKER_MAX_WORKFLOWS
               value: "2"
             # --- kubernetes backend: pipeline steps run as native pods (retires

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,9 +33,25 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"

kubernetes/apps/woodpecker/externalsecret.yaml

@@ -2,7 +2,7 @@ apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
 spec:
   # 10m (was 1h) so an automated rotation (rotate-agent-secret.yaml writes a new
   # value to OpenBao) propagates into the k8s Secret within ~10m without a manual

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/ingress.yaml

@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
   annotations:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,11 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
+  # --- namespace migration, stages 1-2 (issue #433) ---
+  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
+  #
+  # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
+  # not here: it must exist before the `infrastructure-crds` KS applies the
+  # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
   - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/networkpolicy.yaml

@@ -1,20 +1,23 @@
 # Secret rotation: the review-bot and claude-oauth rotators (openbao ns) reach
 # the Woodpecker SERVER to upsert org-level CI secrets (review_bot_token,
 # claude_code_oauth_token). They use the in-cluster Service
-# (woodpecker.apps.svc:8000) — the public ci.renner.uno name hairpins off the
-# ingress LB from a pod. The apps default-deny-ingress only allows same-ns /
-# host / a few named sources, so without this the rotator's connect to :8000 is
+# (woodpecker.woodpecker.svc:8000) — the public ci.renner.uno name hairpins off the
+# ingress LB from a pod. This namespace's baseline
+# (network-policies/woodpecker-ns.yaml) admits only same-namespace, monitoring,
+# host/remote-node and ICMP, so without this the rotator's connect to :8000 is
 # dropped. Scoped to the woodpecker SERVER pod :8000 and the two rotator labels;
 # the egress halves are rotator-review-bot-egress + rotator-claude-oauth-egress
 # in network-policies/openbao-ns.yaml.
 #
-# This is additive to the apps-ns baseline allows (allow-ingress.yaml selects all
-# apps pods), so the server's existing ingress — Traefik, agents — is unaffected.
+# Cilium unions allow rules, so this composes with the baseline and with the
+# sibling policies below (traefik, CI steps) rather than replacing them. Note
+# allow-ingress.yaml does NOT apply here — it is hard-scoped to `namespace:
+# apps`, which is why traefik needs its own rule in this file now.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
   name: woodpecker-allow-rotators
-  namespace: apps
+  namespace: woodpecker
 spec:
   endpointSelector:
     matchLabels:
@@ -34,3 +37,25 @@ spec:
         - ports:
             - port: "8000"
               protocol: TCP
+---
+# Pipeline step pods call the server back on :8000 to stream logs and report
+# status. Deferred from stage 1 on purpose, and scoped to the SERVER pod rather
+# than granted on the namespace baseline.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-ci-steps
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker-ci
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/pdb.yaml

@@ -15,7 +15,7 @@ apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server

kubernetes/apps/woodpecker/rbac.yaml

@@ -17,7 +17,7 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -60,4 +60,7 @@ roleRef:
 subjects:
   - kind: ServiceAccount
     name: woodpecker-agent
-    namespace: apps
+    # The agent's SA lives in `woodpecker`; this Role/RoleBinding stays in
+    # `woodpecker-ci`, because that is the namespace it grants pod
+    # create/delete in. Cross-namespace subject, deliberately.
+    namespace: woodpecker

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster:

kubernetes/apps/woodpecker/secret.example.yaml

@@ -14,7 +14,7 @@ apiVersion: v1
 kind: Secret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 type: Opaque

kubernetes/apps/woodpecker/server.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server
@@ -84,7 +84,7 @@ spec:
             - name: WOODPECKER_DATABASE_DRIVER
               value: postgres
             - name: WOODPECKER_DATABASE_DATASOURCE
-              value: "host=wp-pg-rw.apps.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
+              value: "host=wp-pg-rw.woodpecker.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
           # AGENT_SECRET + GITEA_CLIENT + GITEA_SECRET come from the SOPS secret.
           envFrom:
             - secretRef:

kubernetes/apps/woodpecker/service.yaml

@@ -2,13 +2,13 @@ apiVersion: v1
 kind: Service
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 spec:
   # Flipped ExternalName -> ClusterIP: ci.renner.uno (via the Ingress, by name)
   # and the in-cluster agent both target this Service now. http:8000 serves the
-  # UI/API; grpc:9000 is how the agent (woodpecker.apps.svc:9000) reaches the
+  # UI/API; grpc:9000 is how the agent (woodpecker.woodpecker.svc:9000) reaches the
   # server. Selects the server pod only (not the agent).
   type: ClusterIP
   selector:

kubernetes/apps/wordpress/deployment.yaml

@@ -0,0 +1,115 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wordpress
+  namespace: wordpress
+  annotations:
+    secret.reloader.stakater.com/reload: wordpress-secrets
+spec:
+  replicas: 1
+  strategy:
+    # RWO volume, single writer — the rolling default would deadlock on the
+    # new pod waiting for a volume the old pod still holds.
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: wordpress
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: wordpress
+    spec:
+      securityContext:
+        # wp-content must be writable by Apache (www-data, uid/gid 33) for
+        # media uploads and plugin installs.
+        fsGroup: 33
+      containers:
+        - name: wordpress
+          image: wordpress:7.0.4-php8.3-apache
+          ports:
+            - name: http
+              containerPort: 80
+          env:
+            # The operator names the base Service after the MariaDB resource;
+            # the CRD's separate primaryService/secondaryService fields are the
+            # suffixed variants for replicated setups, which this is not
+            # (replicas: 1). Not verified against a running cluster — if it is
+            # wrong the pod fails loudly on connect at startup rather than
+            # degrading quietly. Confirm with `kubectl get svc -n wordpress`
+            # on first deploy.
+            - name: WORDPRESS_DB_HOST
+              value: wordpress-mariadb.wordpress.svc.cluster.local:3306
+            - name: WORDPRESS_DB_NAME
+              value: wordpress
+            - name: WORDPRESS_DB_USER
+              value: wordpress
+            - name: WORDPRESS_DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  # Generated by the operator from the MariaDB CR — never
+                  # stored in Git or OpenBao.
+                  name: wordpress-mariadb-app
+                  key: password
+            - name: WORDPRESS_CONFIG_EXTRA
+              value: |
+                define('WP_HOME', 'https://www.renner.uno');
+                define('WP_SITEURL', 'https://www.renner.uno');
+                define('FORCE_SSL_ADMIN', true);
+                /* Traefik terminates TLS; without this WordPress sees plain
+                   HTTP and builds http:// URLs into every page and redirect. */
+                if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https') {
+                    $_SERVER['HTTPS'] = 'on';
+                }
+                /* Core lives in the image, not on the volume — the updater
+                   would only fail against a read-only docroot. Plugin and
+                   theme installs are unaffected. */
+                define('WP_AUTO_UPDATE_CORE', false);
+                /* No add_filter() here. WORDPRESS_CONFIG_EXTRA is eval'd inside
+                   wp-config.php, long before wp-settings.php loads the plugin
+                   API, so any add_filter call is a fatal error on every request
+                   — which is exactly what happened on first deploy. xmlrpc is
+                   handled at the ingress instead, by routing /xmlrpc.php
+                   through the Authentik gate. */
+          envFrom:
+            # The eight WordPress salts. Rotating them in OpenBao invalidates
+            # every session — which is the intended lever, not a side effect.
+            - secretRef:
+                name: wordpress-secrets
+          volumeMounts:
+            - name: content
+              mountPath: /var/www/html/wp-content
+          readinessProbe:
+            httpGet:
+              path: /
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: http
+            initialDelaySeconds: 60
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 50m
+              memory: 256Mi
+            limits:
+              memory: 768Mi
+      volumes:
+        - name: content
+          persistentVolumeClaim:
+            claimName: wordpress-content
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wordpress
+  namespace: wordpress
+spec:
+  selector:
+    app.kubernetes.io/name: wordpress
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/wordpress/externalsecret.yaml

@@ -0,0 +1,46 @@
+# The eight WordPress salts (AUTH_KEY, SECURE_AUTH_KEY, LOGGED_IN_KEY,
+# NONCE_KEY, AUTH_SALT, SECURE_AUTH_SALT, LOGGED_IN_SALT, NONCE_SALT).
+# Expected keys in OpenBao at homelab/apps/wordpress, named exactly as the
+# image's env vars: WORDPRESS_AUTH_KEY, WORDPRESS_SECURE_AUTH_KEY, ...
+#
+# Deliberately NOT carried over from the old wp-config.php: the Hetzner host
+# is being decommissioned and its salts have been sitting on a VM whose
+# neighbour site was compromised this month. New salts cost one logout.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wordpress-secrets
+  namespace: wordpress
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wordpress-secrets
+  dataFrom:
+    - extract:
+        key: apps/wordpress
+---
+# S3 credentials for the MariaDB Backup.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: wordpress
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
+    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
+    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
+    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
+    # written, and repoint this here (the value is the same, so it's a drop-in
+    # swap).
+    - extract:
+        key: seed/paperless/hetzner-creds

kubernetes/apps/wordpress/gate-service.yaml

@@ -0,0 +1,15 @@
+# A plain Ingress cannot point at a Service in another namespace, and the
+# Authentik outposts all live in `authentik`. Same workaround as
+# zone-configurator and paperless: mirror the outpost Service in as an
+# ExternalName.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-wordpress-gate
+  namespace: wordpress
+spec:
+  type: ExternalName
+  externalName: ak-outpost-wordpress-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000

kubernetes/apps/wordpress/ingress.yaml

@@ -0,0 +1,110 @@
+# Two routers on one host. The site itself must stay readable by anyone --
+# it is a family blog -- while the editing surface sits behind Authentik.
+#
+# Why not one gated Ingress with skip_path_regex, the way paperless-gate
+# carves out /static/: that works when the PUBLIC paths are enumerable. Here
+# it is the reverse -- every post URL is public and only two paths are gated
+# -- which would need a negative lookahead. The outpost is Go, and RE2 has no
+# lookahead. So the split happens at the router instead.
+#
+# Traefik orders routers by rule length, so PathPrefix(`/wp-admin`) is
+# evaluated before PathPrefix(`/`); no explicit priority is needed.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wordpress
+  namespace: wordpress
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: www.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: wordpress
+                port:
+                  number: 80
+  tls:
+    - secretName: wordpress-tls
+      hosts:
+        - www.renner.uno
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wordpress-admin
+  namespace: wordpress
+  labels:
+    # Same label as the public Ingress on purpose. external-dns-unifi runs
+    # with --label-filter=dns.renner.uno/public!=true, so an unlabelled
+    # Ingress on this host would quietly gain a UDM A-record pointing LAN
+    # clients straight at the Traefik VIP -- a second, unintended path to
+    # the same host that skips the public route entirely.
+    dns.renner.uno/public: "true"
+  # No cert-manager annotation and no tls block: the public Ingress above
+  # already owns the certificate for this host. Declaring it twice would
+  # create two Certificates racing for the same wordpress-tls Secret.
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: www.renner.uno
+      http:
+        paths:
+          # The outpost's own OAuth leg. Without this the gate is a dead end,
+          # not a gate: hitting /wp-admin redirects to
+          # /outpost.goauthentik.io/start on THIS host, which otherwise falls
+          # through to the public `/` router and lands on WordPress's 404
+          # page — verified exactly that way on first deploy. paperless needs
+          # the same route and has it as a separate Ingress on auth.renner.uno
+          # (apps/authentik/paperless-gate-ingress.yaml); here the flow stays
+          # on www.renner.uno, so it belongs in this rule set.
+          #
+          # It is not an authentication hole: these endpoints ARE the login
+          # flow, and the outpost serves them itself.
+          - path: /outpost.goauthentik.io
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000
+          # Covers admin-ajax.php too. That is only safe because the gift
+          # registry plugin -- the sole thing on this site that ever emitted
+          # an ajaxurl -- is being dropped in the migration. Verified against
+          # the live site: 121 admin-ajax hits in the current log window, not
+          # one of them a 200.
+          - path: /wp-admin
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000
+          - path: /wp-login.php
+            pathType: Exact
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000
+          # xmlrpc.php would otherwise be served by the public `/` prefix
+          # router — an earlier version of this file claimed it was "blocked
+          # at the ingress" while nothing blocked it at all. There is no deny
+          # verb in an Ingress, so it goes through the gate instead: anonymous
+          # callers get Authentik, not the XML-RPC endpoint. Nothing on this
+          # site uses it (no Jetpack, no mobile app, no pingbacks worth
+          # keeping), so no legitimate caller is affected.
+          - path: /xmlrpc.php
+            pathType: Exact
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000

kubernetes/apps/wordpress/kustomization.yaml

@@ -0,0 +1,9 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - mariadb.yaml
+  - pvc.yaml
+  - deployment.yaml
+  - gate-service.yaml
+  - ingress.yaml

kubernetes/apps/wordpress/mariadb.yaml

@@ -0,0 +1,86 @@
+# The database for www.renner.uno. Named `wordpress-mariadb`, not `wp-*`:
+# `wp-pg` in the woodpecker namespace is Woodpecker's Postgres, and reusing
+# the prefix for an unrelated engine invites exactly the wrong guess.
+#
+# Mirrors the CNPG convention: the operator lives in `database`, the database
+# resource lives with its consumer (ak-pg in authentik, im-pg in immich).
+apiVersion: k8s.mariadb.com/v1alpha1
+kind: MariaDB
+metadata:
+  name: wordpress-mariadb
+  namespace: wordpress
+  annotations:
+    # Data guard — a Kustomization rebuild must never take the database with it.
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  # generate: true lets the operator create both Secrets on first reconcile.
+  # Neither password is ever typed by a human or parked in OpenBao, so there
+  # is nothing to rotate by hand and nothing to leak through Git.
+  rootPasswordSecretKeyRef:
+    name: wordpress-mariadb-root
+    key: password
+    generate: true
+  username: wordpress
+  passwordSecretKeyRef:
+    name: wordpress-mariadb-app
+    key: password
+    generate: true
+  database: wordpress
+  port: 3306
+  replicas: 1
+
+  storage:
+    size: 5Gi
+    storageClassName: ceph-block
+  # The source database is 527 KB. 5Gi is not a sizing estimate, it is the
+  # smallest allocation that leaves room for binlogs and a decade of posts.
+
+  myCnf: |
+    [mariadb]
+    bind-address=*
+    default_storage_engine=InnoDB
+    innodb_buffer_pool_size=256M
+    # WordPress writes utf8mb4 throughout; the source dump carries
+    # utf8mb4_unicode_520_ci on every table that survives the migration.
+    character_set_server=utf8mb4
+    collation_server=utf8mb4_unicode_520_ci
+    max_allowed_packet=64M
+
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 1Gi
+---
+# Logical backup to Hetzner Object Storage. Velero only captures PVC data,
+# which for a running database is crash-consistent, not transaction-consistent
+# — that is a snapshot, not a backup you would want to restore from.
+apiVersion: k8s.mariadb.com/v1alpha1
+kind: Backup
+metadata:
+  name: wordpress-mariadb-backup
+  namespace: wordpress
+spec:
+  mariaDbRef:
+    name: wordpress-mariadb
+  schedule:
+    # 02:45, ahead of the Velero window that starts at 03:15.
+    cron: "45 2 * * *"
+    suspend: false
+  maxRetention: 720h # 30d, matching the CNPG ObjectStore retentionPolicy
+  compression: gzip
+  storage:
+    s3:
+      bucket: renner-uno-k8s-mariadb
+      prefix: wordpress
+      endpoint: fsn1.your-objectstorage.com
+      region: fsn1
+      accessKeyIdSecretKeyRef:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKeySecretKeyRef:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      tls:
+        enabled: true

kubernetes/apps/wordpress/pvc.yaml

@@ -0,0 +1,35 @@
+# Only wp-content is persisted, not the whole docroot. WordPress core comes
+# from the pinned image, so upgrading the site means bumping a tag in Git
+# rather than clicking "update" in wp-admin — which is the entire point of
+# moving off the Hetzner VM, where core was a mutable pile of PHP on disk.
+#
+# Consequence to be aware of: WordPress's own core updater will fail against
+# a read-only core. Plugin and theme installs still work, because those land
+# in wp-content.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: wordpress-content
+  namespace: wordpress
+  labels:
+    # The Velero Schedule selects ON this label rather than excluding the
+    # database PVC by one the mariadb-operator is merely assumed to set. If
+    # that assumption were wrong the exclusion would fail silently and Velero
+    # would snapshot the live database volume; selecting a label this repo
+    # owns cannot fail that way. Same direction as openbao-daily, which picks
+    # its snapshot PVC by app.kubernetes.io/component.
+    app.kubernetes.io/name: wordpress
+    app.kubernetes.io/component: content
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  # RWO is sufficient: a single replica with strategy Recreate. ceph-block is
+  # the cluster default; named explicitly so a default change can't silently
+  # move the media library onto another class.
+  storageClassName: ceph-block
+  resources:
+    requests:
+      # The migrated wp-content is 29 MB compressed, 13 MB of it uploads.
+      storage: 10Gi

kubernetes/clusters/homelab/infrastructure.yaml

@@ -48,6 +48,10 @@ spec:
       kind: HelmRelease
       name: cloudnative-pg # DB operator — seed jobs create databases
       namespace: database
+    - apiVersion: helm.toolkit.fluxcd.io/v2
+      kind: HelmRelease
+      name: mariadb-operator # the MySQL half — WordPress can't use CNPG
+      namespace: database
   # decryption block dropped 2026-07-13: no *.sops.yaml left under this
   # path — secrets come from OpenBao via ESO (MIGRATION-sops-openbao.md).
 ---