fix(netbird-ingress): switch to rootless image, drop all caps (the real fix) #31

2026-06-26T05:10:56Z

k.renner commented

2026-06-26 05:10:56 +00:00

Switch `netbird-ingress` to the rootless image — the actual fix

The kernel-mode routing peer never worked because of two independent walls we hit during validation:

socketLB: it L3-forwards packets to ClusterIP/LB VIPs, but Cilium socketLB only DNATs a pod's own socket connections, not forwarded packets. Confirmed via conntrack on the peer — the forwarded DNS flow stays [UNREPLIED] (src=…→dst=10.2.99.83, no reply), while the pod's own socket to 10.2.99.83/fd99::83 answers fine.
Pod Security: it needs NET_ADMIN/SYS_ADMIN, which PSA baseline rejects (this is what took audiobookshelf down in the sidecar experiment, now reverted).

The rootless image (netbirdio/netbird:v0.73.2-rootless, userspace wireguard-go) fixes both at once — and it's the documented community pattern for running a NetBird routing peer in k8s (writeup, NetBird docs):

The netbird process terminates the tunnel in userspace and reaches services by opening its own sockets → socketLB DNATs those correctly. The exact path our conntrack test already proved works.
No kernel WireGuard → fully unprivileged: drop: ["ALL"], runAsNonRoot, runAsUser: 65532, allowPrivilegeEscalation: false. Satisfies PSA baseline/restricted.

Changes

image → netbirdio/netbird:v0.73.2-rootless; securityContext → capless/non-root; USER/HOME env + fsGroup: 65532 for the state emptyDir.
image-automation policy now tracks vX.Y.Z-rootless tags.
Unchanged: the setup-key secret, the netbird-ingress-egress CNP, and your NetBird-side routes (fd99::/64 + 10.2.99.0/24) and nameserver (renner.uno → fd99::83/10.2.99.83). Those finally work now that forwarding is userspace.

Validate (from the laptop, after roll)

kubectl get pods -n ingress -l app=netbird-ingress     # capless pod Ready
dig audiobooks.renner.uno                              # resolves, fast (no 2s timeout)
curl -sI https://audiobooks.renner.uno                 # 200 over the mesh

This validates the original standalone-routing-peer architecture; the sidecar/proxy/PSA-relaxation detours are off the table.

🤖 Generated with Claude Code

## Switch `netbird-ingress` to the rootless image — the actual fix The kernel-mode routing peer never worked because of **two** independent walls we hit during validation: 1. **socketLB:** it L3-forwards packets to ClusterIP/LB VIPs, but Cilium socketLB only DNATs a pod's **own socket** connections, not forwarded packets. Confirmed via conntrack on the peer — the forwarded DNS flow stays `[UNREPLIED]` (`src=…→dst=10.2.99.83`, no reply), while the pod's *own* socket to `10.2.99.83`/`fd99::83` answers fine. 2. **Pod Security:** it needs `NET_ADMIN`/`SYS_ADMIN`, which PSA `baseline` rejects (this is what took audiobookshelf down in the sidecar experiment, now reverted). The **rootless image** (`netbirdio/netbird:v0.73.2-rootless`, userspace `wireguard-go`) fixes both at once — and it's the documented community pattern for running a NetBird routing peer in k8s ([writeup](https://johanneskueber.com/posts/2026-05-25-netbird-routing-peer/), [NetBird docs](https://docs.netbird.io/how-to/kubernetes-operator)): - The netbird process terminates the tunnel **in userspace** and reaches services by opening its **own sockets** → socketLB DNATs those correctly. The exact path our conntrack test already proved works. - No kernel WireGuard → **fully unprivileged**: `drop: ["ALL"]`, `runAsNonRoot`, `runAsUser: 65532`, `allowPrivilegeEscalation: false`. Satisfies PSA `baseline`/`restricted`. ### Changes - image → `netbirdio/netbird:v0.73.2-rootless`; securityContext → capless/non-root; `USER`/`HOME` env + `fsGroup: 65532` for the state emptyDir. - image-automation policy now tracks `vX.Y.Z-rootless` tags. - **Unchanged:** the setup-key secret, the `netbird-ingress-egress` CNP, and your NetBird-side routes (`fd99::/64` + `10.2.99.0/24`) and nameserver (`renner.uno → fd99::83/10.2.99.83`). Those finally work now that forwarding is userspace. ### Validate (from the laptop, after roll) ``` kubectl get pods -n ingress -l app=netbird-ingress # capless pod Ready dig audiobooks.renner.uno # resolves, fast (no 2s timeout) curl -sI https://audiobooks.renner.uno # 200 over the mesh ``` This validates the original standalone-routing-peer architecture; the sidecar/proxy/PSA-relaxation detours are off the table. 🤖 Generated with Claude Code

k.renner added 1 commit 2026-06-26 05:10:57 +00:00

fix(netbird-ingress): switch to rootless image, drop all caps 9fc44d8688

The kernel-mode routing peer could not deliver mesh traffic to cluster
services: it L3-forwards packets to ClusterIP/LB VIPs, but Cilium
socket-LB only DNATs a pod's OWN socket connections, not forwarded
packets — so the VIP destination is blackholed (confirmed via conntrack:
the flow stays [UNREPLIED] while the pod's own socket to the same VIP
answers). It also needed NET_ADMIN/SYS_ADMIN, which Pod Security
baseline forbids in app namespaces.

The rootless image (userspace wireguard-go) fixes both: the netbird
process terminates the tunnel in userspace and reaches services by
opening its OWN sockets — which socket-LB DNATs correctly — and needs
no kernel WireGuard, so it runs fully unprivileged (drop ALL caps,
runAsNonRoot:65532). This is the documented community pattern for
running a NetBird routing peer in Kubernetes.

- image -> netbirdio/netbird:v0.73.2-rootless
- securityContext -> capless, runAsNonRoot, no privilege escalation
- add USER/HOME + fsGroup so the state emptyDir is writable by 65532
- image-automation policy now tracks vX.Y.Z-rootless tags
- existing setup-key secret, CNP, and NetBird-side routes/nameserver
  are unchanged — they finally work now that forwarding is userspace

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 5263146222 into master

2026-06-26 05:19:43 +00:00

k.renner referenced this issue from a commit

2026-06-26 05:19:43 +00:00

Merge pull request 'fix(netbird-ingress): switch to rootless image, drop all caps (the real fix)' (#31) from netbird-ingress-rootless into master

k.renner referenced this pull request

2026-07-03 12:34:26 +00:00

edge-host: Mischer-Stellung Heizkreise 2/3 (DIDs 476/477) #104

Sign in to join this conversation.