feat(netbird): retire the reverse proxy on the management host #965

2026-09-25T09:18:25Z

claude-bot commented

2026-09-25 09:18:25 +00:00

Why

The reverse proxy on netbird.renner.uno has served nothing since 2026-09-07. The controller switched every service to TLS passthrough, which runs on the proxy's own :443 listener, but the host's Traefik forwards HostSNI(*) to the proxy's :8443 main listener ("unknown domain"). proxy.renner.uno no longer points at this host, so public traffic is served only by proxy-1/proxy-2.

Its watchdog failed every probe round (home/immich = 000; last healthy round 2026-09-07 12:53) and has run docker restart netbird-proxy every ~3 minutes since. Every restart registers a new proxy peer and pushes a network-map update to the whole mesh: the cluster ingress peers hold ~10 stale proxy-* identities and get a map update every 1–2 minutes.

Found while investigating intermittent external ABS streaming. This PR is not shown to be the cause of that; it removes a confirmed restart loop and its mesh churn.

Changes

netbird-server role: removes the proxy service + Traefik TCP passthrough, proxy.env, the proxy vars and the proxy handler. compose up -d --remove-orphans removes the container. One-off tasks stop/disable both watchdogs (URL probe + flood) before the container goes, then delete their units/scripts, the netbird_netbird_proxy_{certs,data,state} volumes, and /opt/netbird/{proxy.env,proxy,proxy-state}.
install-netbird-server.yml no longer applies netbird-proxy-watchdog. deploy-netbird-server no longer triggers on that role.
Comments that described the management host as a proxy cluster member now describe the proxy hosts only (netbird-proxy defaults/compose, watchdog defaults, install/rotate plays).

Deploy notes

The compose-file change fires the existing Restart netbird containers handler (down + up) on the management host: a short management/signal/relay restart, the same as any compose change. Established WireGuard tunnels stay up.
The watchdog-role comment edits also trigger deploy-netbird-proxy (serial: 1, one host at a time). Templates are unchanged, so it should report no changes on proxy-1/proxy-2.
vault_netbird_proxy_token stays; proxy-1/proxy-2 use it.

Verification after merge

docker ps on the management host: no netbird-proxy. systemctl list-timers: no watchdog timer.
netbird status -d in a netbird-ingress pod: no new proxy-* peers appearing, map updates stop every 1–2 min.
curl audiobooks.renner.uno/healthcheck against each proxy.renner.uno address → 200.

Local check: ansible-playbook --syntax-check on the server play passes; yamllint/ansible-lint are not installed locally.

🤖 Generated with Claude Code

## Why The reverse proxy on `netbird.renner.uno` has served nothing since 2026-09-07. The controller switched every service to TLS passthrough, which runs on the proxy's own `:443` listener, but the host's Traefik forwards `HostSNI(*)` to the proxy's `:8443` main listener ("unknown domain"). `proxy.renner.uno` no longer points at this host, so public traffic is served only by proxy-1/proxy-2. Its watchdog failed every probe round (`home`/`immich` = `000`; last healthy round 2026-09-07 12:53) and has run `docker restart netbird-proxy` every ~3 minutes since. Every restart registers a new proxy peer and pushes a network-map update to the whole mesh: the cluster ingress peers hold ~10 stale `proxy-*` identities and get a map update every 1–2 minutes. Found while investigating intermittent external ABS streaming. This PR is **not** shown to be the cause of that; it removes a confirmed restart loop and its mesh churn. ## Changes - **netbird-server role**: removes the `proxy` service + Traefik TCP passthrough, `proxy.env`, the proxy vars and the proxy handler. `compose up -d --remove-orphans` removes the container. One-off tasks stop/disable both watchdogs (URL probe + flood) *before* the container goes, then delete their units/scripts, the `netbird_netbird_proxy_{certs,data,state}` volumes, and `/opt/netbird/{proxy.env,proxy,proxy-state}`. - **install-netbird-server.yml** no longer applies `netbird-proxy-watchdog`. `deploy-netbird-server` no longer triggers on that role. - Comments that described the management host as a proxy cluster member now describe the proxy hosts only (netbird-proxy defaults/compose, watchdog defaults, install/rotate plays). ## Deploy notes - The compose-file change fires the existing `Restart netbird containers` handler (`down` + `up`) on the management host: a short management/signal/relay restart, the same as any compose change. Established WireGuard tunnels stay up. - The watchdog-role comment edits also trigger `deploy-netbird-proxy` (serial: 1, one host at a time). Templates are unchanged, so it should report no changes on proxy-1/proxy-2. - `vault_netbird_proxy_token` stays; proxy-1/proxy-2 use it. ## Verification after merge - `docker ps` on the management host: no `netbird-proxy`. `systemctl list-timers`: no watchdog timer. - `netbird status -d` in a `netbird-ingress` pod: no new `proxy-*` peers appearing, map updates stop every 1–2 min. - `curl` `audiobooks.renner.uno/healthcheck` against each `proxy.renner.uno` address → 200. Local check: `ansible-playbook --syntax-check` on the server play passes; yamllint/ansible-lint are not installed locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-25 09:18:25 +00:00

feat(netbird): retire the reverse proxy on the management host

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

adbfd3bfc6

The public edge is proxy-1/proxy-2; proxy.renner.uno has not pointed at
netbird.renner.uno since the netcup fleet took over. The proxy that kept
running on the management host has served nothing since the controller
moved every service to TLS passthrough (2026-09-07): passthrough routes
live on the proxy's own :443 listener, while the host's Traefik forwards
HostSNI(*) to its :8443 main listener, which answers "unknown domain".

Its watchdog therefore failed every probe round and ran `docker restart`
every ~3 minutes since 2026-09-07 12:53 (last healthy round). Each
restart registered a fresh proxy peer and pushed a network-map update to
every peer in the mesh — ~10 stale proxy-* identities per ingress peer
and a map update every 1-2 minutes.

- netbird-server: drop the proxy service, its Traefik TCP passthrough,
  proxy.env and the proxy vars; `compose up --remove-orphans` removes
  the container, and one-off tasks stop/remove both watchdogs and delete
  the proxy volumes, config and state.
- install-netbird-server.yml no longer applies netbird-proxy-watchdog;
  its pipeline no longer triggers on that role.
- Comments that described the management host as a cluster member now
  describe the proxy hosts only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review-bot reviewed

2026-09-25 09:22:14 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit adbfd3bfc681e4543b1e9beed4259e44a575ea7f.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Removes the reverse-proxy service, its watchdogs, and related config/volumes/comments from the management host's netbird-server role, since proxy.renner.uno no longer routes traffic there and the dead proxy was restart-looping and spamming NetBird mesh map updates. The change is well-sequenced (stop/disable watchdogs → docker compose up -d --remove-orphans → drop volumes/files) and the CI path filters, playbook role list, and cross-role comments are all updated consistently. No blocking issues found.

🟡 Suggestions

ansible/roles/netbird-server/tasks/main.yml:121 — the "Remove reverse-proxy volumes" loop targets netbird_netbird_proxy_certs, netbird_netbird_proxy_data, and netbird_netbird_proxy_state. Based on the removed compose file (ansible/roles/netbird-server/templates/docker-compose.yml.j2, top-level volumes: block before this PR), the management host only ever declared a named volume for netbird_proxy_certs — the proxy's /var/lib/netbird state was a bind mount to /opt/netbird/proxy-state (removed separately at line 137), not a Docker volume, and no ..._data volume existed anywhere in the codebase (only the separate netbird-proxy role used on proxy-1/proxy-2 defines netbird_proxy_certs/netbird_proxy_state as its own project's volumes, which is a different host/compose project). The _data/_state entries in this loop are therefore dead lookups on this host — harmless because the task guards with docker volume ls -q --filter before removing, but worth trimming to just netbird_netbird_proxy_certs (or leaving a note if you know of a volume name I'm missing) to avoid implying state that never existed here.

Tests

No automated tests exist for this Ansible repo beyond syntax/lint (confirmed via the PR description: ansible-playbook --syntax-check passed locally). That matches the repo's existing testing posture for infra roles — nothing further to add here. The PR's own "Verification after merge" checklist (no netbird-proxy container, no watchdog timer, no new proxy-* peers, proxy.renner.uno healthcheck) is a reasonable manual verification plan for a change like this.

Recommendation

Looks reasonable — no blocking issues; the comment above is optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `adbfd3bfc681e4543b1e9beed4259e44a575ea7f`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Removes the reverse-proxy service, its watchdogs, and related config/volumes/comments from the management host's `netbird-server` role, since `proxy.renner.uno` no longer routes traffic there and the dead proxy was restart-looping and spamming NetBird mesh map updates. The change is well-sequenced (stop/disable watchdogs → `docker compose up -d --remove-orphans` → drop volumes/files) and the CI path filters, playbook role list, and cross-role comments are all updated consistently. No blocking issues found. ## 🟡 Suggestions - `ansible/roles/netbird-server/tasks/main.yml:121` — the "Remove reverse-proxy volumes" loop targets `netbird_netbird_proxy_certs`, `netbird_netbird_proxy_data`, and `netbird_netbird_proxy_state`. Based on the removed compose file (`ansible/roles/netbird-server/templates/docker-compose.yml.j2`, top-level `volumes:` block before this PR), the management host only ever declared a named volume for `netbird_proxy_certs` — the proxy's `/var/lib/netbird` state was a bind mount to `/opt/netbird/proxy-state` (removed separately at line 137), not a Docker volume, and no `..._data` volume existed anywhere in the codebase (only the separate `netbird-proxy` role used on proxy-1/proxy-2 defines `netbird_proxy_certs`/`netbird_proxy_state` as *its own* project's volumes, which is a different host/compose project). The `_data`/`_state` entries in this loop are therefore dead lookups on this host — harmless because the task guards with `docker volume ls -q --filter` before removing, but worth trimming to just `netbird_netbird_proxy_certs` (or leaving a note if you know of a volume name I'm missing) to avoid implying state that never existed here. ## Tests No automated tests exist for this Ansible repo beyond syntax/lint (confirmed via the PR description: `ansible-playbook --syntax-check` passed locally). That matches the repo's existing testing posture for infra roles — nothing further to add here. The PR's own "Verification after merge" checklist (no `netbird-proxy` container, no watchdog timer, no new `proxy-*` peers, `proxy.renner.uno` healthcheck) is a reasonable manual verification plan for a change like this. ## Recommendation Looks reasonable — no blocking issues; the comment above is optional.

k.renner merged commit 48bbe511b8 into master

2026-09-25 09:26:20 +00:00

k.renner referenced this issue from a commit

2026-09-25 09:26:22 +00:00

Merge pull request 'feat(netbird): retire the reverse proxy on the management host' (#965) from feat/netbird-retire-vps-proxy into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#965