feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)

Pins the controller image by digest. The registry tag is the build
date and pipeline 8452 rebuilt today's tag in place, so the tag alone
cannot carry a new build to the cluster; the digest can.

Merge only after the Traefik change from 1/2 has reconciled: from this
build on every service target sends a PROXY protocol v2 header, and
Traefik drops such connections unless websecure trusts the pod CIDRs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ansible/roles/netbird-proxy-watchdog/defaults/main.yml

@@ -32,8 +32,13 @@ netbird_watchdog_fail_threshold: 3
 
 # Minimum seconds between watchdog-issued restarts, so a genuinely broken
 # backend can't put the proxy into a restart loop. Shared by both the URL-probe
-# watchdog and the flood watchdog below.
-netbird_watchdog_cooldown_seconds: 600
+# watchdog and the flood watchdog below. One minute: the route-manager wedge
+# recurs on every routing-peer loss, and rolling node reboots lose one peer
+# every few minutes; a suppressed restart there leaves the wedge standing
+# until the URL probe's three rounds fire (3.5 min of 000 on every domain,
+# measured 2026-09-07). A minute still caps a loop at one restart per
+# probe interval.
+netbird_watchdog_cooldown_seconds: 60
 
 # Log fingerprint of the reverse-proxy route-manager wedge, matched as a plain
 # substring in the proxy's docker logs by the flood watchdog. On this build the

ci/images/netbird-proxy-services/controller.py

@@ -6,7 +6,9 @@ For each host on such an Ingress the NetBird management gets one reverse-proxy
 service in TLS-passthrough mode on :443, targeting Traefik's ClusterIP through a
 subnet resource that covers the cluster's service CIDR. SNI and Host reach
 Traefik untouched, the proxy terminates nothing, and no domain resource is
-involved -- the routing peers stay ordinary IP routers.
+involved -- the routing peers stay ordinary IP routers. The target sends a
+PROXY protocol v2 header so Traefik (and CrowdSec behind its access log) sees
+the WAN client address instead of the routing peer's pod IP.
 
 Ownership is derived from NetBird itself, not from local state: a service is
 managed iff its mode is "tls" and its target is this controller's subnet
@@ -194,9 +196,12 @@ def ensure_policy(nb, resource, source_group):
 
 
 def service_body(host, resource, target_ip):
+    # proxy_protocol: PROXY v2 header carries the WAN client address to
+    # Traefik (proxyProtocol.trustedIPs on websecure); TCP/TLS modes only.
     return {"name": host, "domain": host, "mode": "tls", "listen_port": LISTEN_PORT, "enabled": True,
             "targets": [{"target_type": "subnet", "target_id": resource, "protocol": "tcp",
-                         "host": target_ip, "port": 443, "enabled": True}]}
+                         "host": target_ip, "port": 443, "enabled": True,
+                         "options": {"proxy_protocol": True}}]}
 
 
 def is_managed(svc, resource):
@@ -208,7 +213,8 @@ def target_matches(svc, target_ip):
         return False
     ts = svc.get("targets") or []
     return len(ts) == 1 and ts[0].get("host") == target_ip and ts[0].get("port") == 443 \
-        and ts[0].get("protocol") == "tcp" and ts[0].get("enabled")
+        and ts[0].get("protocol") == "tcp" and ts[0].get("enabled") \
+        and (ts[0].get("options") or {}).get("proxy_protocol") is True
 
 
 def reconcile():

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -31,22 +31,12 @@
 # serves read-only decision streams to a valid bouncer key and nothing else.
 # The admin surface is cscli, in-cluster. If the source address ever becomes
 # deterministic, an ipAllowList can be added on top.
----
+#
 # The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
 # fallback and no second port; 8080 is the TLS port. Its certificate comes
 # from the chart's own self-signed CA, so Traefik needs that CA to verify the
-# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
-# what the Service resolves to.
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: crowdsec-lapi
-  namespace: crowdsec
-spec:
-  serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAs:
-    - secret: crowdsec-root-secret
----
+# backend; the ServersTransport carrying it is a traefik.io CRD consumer and
+# lives in post-crds/crowdsec-lapi-serverstransport.yaml.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:

kubernetes/infrastructure/kustomization.yaml

@@ -18,6 +18,7 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
+  - netbird-proxy-services/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/netbird-proxy-services/deployment.yaml

@@ -0,0 +1,106 @@
+# Reconciles NetBird reverse-proxy services from Ingresses that carry
+# dns.renner.uno/public: one TLS-passthrough service per host on :443,
+# targeting Traefik's ClusterIP through a subnet resource over the service
+# CIDR (ci/images/netbird-proxy-services/controller.py).
+#
+# openbao namespace, like the rotators: its default-deny already allows egress
+# to OpenBao, DNS, the apiserver and `world` (netbird.renner.uno), and the
+# NetBird PAT is read from OpenBao on every pass through the
+# netbird-proxy-services role — policies/netbird-proxy-services.hcl, applied
+# once by hand like the rotator roles. Until that role exists the pod logs an
+# auth error each minute and does nothing.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: netbird-proxy-services
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: netbird-proxy-services
+---
+# Read-only, cluster-wide: public Ingresses live in many namespaces, the
+# Traefik Service in ingress/.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: netbird-proxy-services-read
+rules:
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: netbird-proxy-services-read
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: netbird-proxy-services-read
+subjects:
+  - kind: ServiceAccount
+    name: netbird-proxy-services
+    namespace: openbao
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: netbird-proxy-services
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: netbird-proxy-services
+spec:
+  replicas: 1
+  # One writer against the management API at a time.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: netbird-proxy-services
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: netbird-proxy-services
+    spec:
+      serviceAccountName: netbird-proxy-services
+      securityContext:
+        runAsNonRoot: true
+        # Numeric, or the kubelet refuses the image's named user with
+        # "cannot verify user is non-root". The script needs no home and no
+        # writable path, so nobody/nogroup is enough.
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: controller
+          # Pinned by digest: the date tag is rebuilt on every push of the
+          # image directory, so the tag alone cannot express a new build.
+          image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:0d266b0288aec692c698ff5c05f76c826e247bfef365e0306a0b9a6541ebf180
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: NB_MANAGEMENT_URL
+              value: https://netbird.renner.uno
+            - name: BAO_ADDR
+              value: http://openbao.openbao.svc.cluster.local:8200
+            - name: BAO_ROLE
+              value: netbird-proxy-services
+            # Must match cluster.network.serviceSubnets in talos/patches/cluster.yaml.
+            - name: SERVICE_CIDR
+              value: "10.96.0.0/12"
+            - name: TRAEFIK_SERVICE
+              value: ingress/traefik
+            - name: INTERVAL_SECONDS
+              value: "60"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi

kubernetes/infrastructure/netbird-proxy-services/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -31,7 +31,5 @@ resources:
   - traefik-crowdsec-middleware.yaml
   - traefik-rate-limit-middleware.yaml
   - traefik-security-headers-middleware.yaml
-  # Phase 1 of a two-Kustomization move: also still in crowdsec/lapi-ingress.yaml
-  # until this layer has applied it once, so prune never runs ahead of apply.
   - crowdsec-lapi-serverstransport.yaml
   - ../network-policies/

kubernetes/infrastructure/traefik/helmrelease.yaml

@@ -93,13 +93,17 @@ spec:
         # Real client-IP visibility. The netbird-ingress routing peer
         # re-originates every mesh + WAN-via-proxy connection from its own
         # pod IP (rootless userspace WireGuard), so Traefik's TCP peer for
-        # that traffic is always a pod IP (e.g. fd10:244::2de1) — collapsing
-        # all external clients to one source. The NetBird reverse-proxy
-        # (HTTP mode, on the VPS) sets X-Forwarded-For with the real WAN
-        # client IP; trust the pod CIDRs so Traefik reads it instead. Without
-        # this, a CrowdSec bouncer bans the shared pod IP and 403s everyone
-        # (the 2026-06-27 IPv6 outage). Harmless no-op when no XFF is present
-        # (direct LAN/mesh) — those keep their RemoteAddr.
+        # that traffic is always a pod IP — collapsing all external clients
+        # to one source. The NetBird reverse-proxy services are TLS
+        # passthrough and carry the WAN client address in a PROXY protocol
+        # v2 header (options.proxy_protocol, set by netbird-proxy-services);
+        # X-Forwarded-For stays trusted for HTTP-mode services. Both are
+        # accepted from the pod CIDRs only; connections without a header
+        # (direct LAN/mesh) keep their RemoteAddr.
+        proxyProtocol:
+          trustedIPs:
+            - "10.244.0.0/16"
+            - "fd10:244::/56"
         forwardedHeaders:
           trustedIPs:
             - "10.244.0.0/16"