fix(netbird): direct P2P for proxy→cluster and intra-cluster peers #406

2026-07-25T09:11:51Z

claude-bot commented

2026-07-25 09:11:51 +00:00

Why

Every connection of the k8s netbird-ingress peers currently relays through the single relay on the controller VPS (verified live 2026-07-25: 5/5 connected peers Connection type: Relayed, zero P2P, including intra-cluster peer↔peer). That is why the 2026-07-24 Hetzner DC outage took the public edge down even though the reverse-proxy cluster (proxy-1/proxy-2, netcup) is at a different provider. Cause is structural — three compounding blockers, two of which this PR removes:

The proxy containers ran on the Docker bridge publishing only 443:443, so the embedded NetBird client's WireGuard/ICE UDP sat behind Docker NAT66 on a random port.
The netbird-ingress-egress CNP allows egress UDP only to fixed ports 3478/51820 — ICE checks toward random NAT-mapped ports were dropped (the policy's own comment calls relay-over-443 the designed fallback).
Pod-to-pod between the ingress peers was absent from the allowlist entirely, so even same-cluster peers relayed via Hetzner.

Changes

ansible/roles/netbird-proxy: proxy container moves to network_mode: host. Its ICE host candidates become the hosts' un-NATed public addresses, so the k8s peers reach them outbound-only — no hole punching required. Verified beforehand: pods have working v4 and v6 egress (matters because proxy-1 is v6-only; proxy-2 is dual-stack), and neither proxy host runs a host firewall, so :443 exposure is unchanged from the published port. Stale bridge-v6 comment in the role updated; daemon-level bridge v6 stays for the http-redirect sidecar.
kubernetes/infrastructure/network-policies/ingress-isolation.yaml: two new egress rules — pod-to-pod between app: netbird-ingress peers (no port pin; ICE may use ephemeral candidate ports), and all-ports egress to the three proxy addresses (45.157.177.26, both netcup /128s; the embedded client's UDP port is 51820 by default but not contractual). Sync note added referencing proxy DNS + unifi_nat_hairpin.tf.

The controller relay remains as the designed fallback — this PR just stops it being the load-bearing path for the public edge.

Rollout & verification

Merge auto-triggers the deploy-netbird-proxy Woodpecker pipeline (play runs serial: 1, one cluster member at a time) and Flux applies the CNP.
CNP passed a server-side dry-run against the live CRD; infrastructure kustomization builds.
After rollout: netbird status -d in a netbird-ingress pod should show Connection type: P2P for the proxy peers and sibling ingress peers; the 2026-07-24 failure mode (Hetzner down ⇒ public edge down) then requires both providers to fail.

🤖 Generated with Claude Code

## Why Every connection of the k8s `netbird-ingress` peers currently relays through the single relay on the controller VPS (verified live 2026-07-25: 5/5 connected peers `Connection type: Relayed`, zero P2P, including intra-cluster peer↔peer). That is why the 2026-07-24 Hetzner DC outage took the public edge down even though the reverse-proxy cluster (proxy-1/proxy-2, netcup) is at a different provider. Cause is structural — three compounding blockers, two of which this PR removes: 1. The proxy containers ran on the Docker bridge publishing only `443:443`, so the embedded NetBird client's WireGuard/ICE UDP sat behind Docker NAT66 on a random port. 2. The `netbird-ingress-egress` CNP allows egress UDP only to fixed ports 3478/51820 — ICE checks toward random NAT-mapped ports were dropped (the policy's own comment calls relay-over-443 the designed fallback). 3. Pod-to-pod between the ingress peers was absent from the allowlist entirely, so even same-cluster peers relayed via Hetzner. ## Changes - **`ansible/roles/netbird-proxy`**: proxy container moves to `network_mode: host`. Its ICE host candidates become the hosts' un-NATed public addresses, so the k8s peers reach them **outbound-only** — no hole punching required. Verified beforehand: pods have working v4 **and** v6 egress (matters because proxy-1 is v6-only; proxy-2 is dual-stack), and neither proxy host runs a host firewall, so `:443` exposure is unchanged from the published port. Stale bridge-v6 comment in the role updated; daemon-level bridge v6 stays for the `http-redirect` sidecar. - **`kubernetes/infrastructure/network-policies/ingress-isolation.yaml`**: two new egress rules — pod-to-pod between `app: netbird-ingress` peers (no port pin; ICE may use ephemeral candidate ports), and all-ports egress to the three proxy addresses (45.157.177.26, both netcup /128s; the embedded client's UDP port is 51820 by default but not contractual). Sync note added referencing proxy DNS + `unifi_nat_hairpin.tf`. The controller relay remains as the designed fallback — this PR just stops it being the load-bearing path for the public edge. ## Rollout & verification - Merge auto-triggers the `deploy-netbird-proxy` Woodpecker pipeline (play runs `serial: 1`, one cluster member at a time) and Flux applies the CNP. - CNP passed a server-side dry-run against the live CRD; infrastructure kustomization builds. - After rollout: `netbird status -d` in a `netbird-ingress` pod should show `Connection type: P2P` for the proxy peers and sibling ingress peers; the 2026-07-24 failure mode (Hetzner down ⇒ public edge down) then requires *both* providers to fail. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-25 09:11:51 +00:00

fix(netbird): direct P2P for proxy→cluster and intra-cluster peers

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

23654d8b4b

Every connection of the k8s netbird-ingress peers relays through the
single controller-VPS relay (verified 2026-07-25: 5/5 connected peers
Relayed, zero P2P) — which is why the 2026-07-24 Hetzner DC outage took
the public edge down even though the reverse-proxy cluster runs at a
different provider. Three structural blockers, two fixed here:

- netcup proxy members: run the reverse-proxy container on the host
  network instead of bridge+443:443. Behind the bridge the embedded
  client's WireGuard/ICE UDP sat on a random Docker-NAT66 port that the
  pods' egress policy could never match; on the host network its ICE
  host candidates are the un-NATed public addresses, reachable
  outbound-only from the pods (no hole punching). proxy-2 serves
  v4+v6, proxy-1 v6-only — pods have working egress on both families
  (verified). :443 binds as before; the hosts run no firewall
  (verified), so exposure is unchanged.
- netbird-ingress-egress CNP: allow pod-to-pod between the ingress
  peers (previously absent entirely — even same-cluster peers relayed
  via Hetzner) and any-port UDP-capable egress to the three proxy
  addresses (the embedded client's port is 51820 by default but not
  contractual).

The controller relay remains available as designed fallback. Merging
auto-triggers the deploy-netbird-proxy Woodpecker pipeline (serial: 1,
one member at a time) and Flux applies the CNP.

Verify after rollout: netbird status -d in a netbird-ingress pod should
show 'Connection type: P2P' for the proxy peers and the sibling ingress
peers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

review-bot reviewed

2026-07-25 09:16:09 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 23654d8b4b3ba6cd1b5c071866a469325f56b625.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR moves the netbird-proxy container (proxy-1/proxy-2) from Docker bridge networking to network_mode: host, and adds two egress rules to the netbird-ingress-egress CiliumNetworkPolicy: pod-to-pod between netbird-ingress peers, and all-ports egress to the three proxy hosts' public addresses. Both halves of the change are consistent with each other and with the rest of the repo (the CIDR values match opentofu/unifi_nat_hairpin.tf exactly, the netbird_proxy inventory group only contains proxy-1/proxy-2 so VPS-1's Traefik-SNI :443 sharing is untouched, and no host-firewall role exists on these hosts, supporting the "unchanged exposure" claim in the compose comment). I found no blocking issues.

🟡 Suggestions

kubernetes/infrastructure/network-policies/ingress-isolation.yaml:62-65 — the new toCIDRSet rule (and the pod-to-pod toEndpoints rule above it at line 67-69) allow all protocols and all ports, not just UDP. The stated reason (ICE uses ephemeral candidate ports) only requires leaving the port unpinned — the protocol could still be restricted to UDP via toPorts: [{port: "1", endPort: 65535, protocol: UDP}] (Cilium port ranges), which would stop a compromised netbird-ingress pod from reaching arbitrary TCP ports on the proxy hosts (which, per the PR body, run no host firewall). Not blocking — this is a deliberate, documented trade-off for a small, trusted set of destinations, but worth a second look given the hosts are otherwise unfirewalled.
No test/dry-run artifact (kustomize build output, cilium policy validate, etc.) is included in the PR itself — the body asserts a dry-run and kustomize build were run manually, but that's not independently checkable from the diff alone.

Tests

This is infrastructure config (Ansible + CiliumNetworkPolicy); the repo has no automated test suite for either. Verification is manual, as the PR's own "Rollout & verification" section lays out (netbird status -d should show Connection type: P2P post-rollout). That matches the project's existing norms for this kind of change — reasonable given the constraints.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `23654d8b4b3ba6cd1b5c071866a469325f56b625`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR moves the `netbird-proxy` container (proxy-1/proxy-2) from Docker bridge networking to `network_mode: host`, and adds two egress rules to the `netbird-ingress-egress` CiliumNetworkPolicy: pod-to-pod between `netbird-ingress` peers, and all-ports egress to the three proxy hosts' public addresses. Both halves of the change are consistent with each other and with the rest of the repo (the CIDR values match `opentofu/unifi_nat_hairpin.tf` exactly, the `netbird_proxy` inventory group only contains proxy-1/proxy-2 so VPS-1's Traefik-SNI :443 sharing is untouched, and no host-firewall role exists on these hosts, supporting the "unchanged exposure" claim in the compose comment). I found no blocking issues. ## 🟡 Suggestions - `kubernetes/infrastructure/network-policies/ingress-isolation.yaml:62-65` — the new `toCIDRSet` rule (and the pod-to-pod `toEndpoints` rule above it at line 67-69) allow all protocols and all ports, not just UDP. The stated reason (ICE uses ephemeral candidate ports) only requires leaving the *port* unpinned — the *protocol* could still be restricted to UDP via `toPorts: [{port: "1", endPort: 65535, protocol: UDP}]` (Cilium port ranges), which would stop a compromised `netbird-ingress` pod from reaching arbitrary TCP ports on the proxy hosts (which, per the PR body, run no host firewall). Not blocking — this is a deliberate, documented trade-off for a small, trusted set of destinations, but worth a second look given the hosts are otherwise unfirewalled. - No test/dry-run artifact (kustomize build output, `cilium policy validate`, etc.) is included in the PR itself — the body asserts a dry-run and kustomize build were run manually, but that's not independently checkable from the diff alone. ## Tests This is infrastructure config (Ansible + CiliumNetworkPolicy); the repo has no automated test suite for either. Verification is manual, as the PR's own "Rollout & verification" section lays out (`netbird status -d` should show `Connection type: P2P` post-rollout). That matches the project's existing norms for this kind of change — reasonable given the constraints. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 6d286ac00a into master

2026-07-26 05:35:03 +00:00

k.renner referenced this issue from a commit

2026-07-26 05:35:03 +00:00

Merge pull request 'fix(netbird): direct P2P for proxy→cluster and intra-cluster peers' (#406) from netbird-p2p-direct into master

claude-bot referenced this issue from a commit

2026-07-26 05:41:12 +00:00

hotfix(netbird-proxy): allow non-root :443 bind on the host network

claude-bot referenced this pull request

2026-07-26 05:41:24 +00:00

hotfix(netbird-proxy): allow non-root :443 bind on the host network #409

claude-bot referenced this pull request