fix(ansible): stop rotating netbird_proxy_token — Ansible does not own both ends #632

2026-08-26T10:56:36Z

claude-bot commented

2026-08-26 10:56:36 +00:00

Rotation ran in production. turn_secret and relay_secret are rotated and correct. proxy_token was in the same set and should never have been — rotating it locked every proxy out of the management server. Recovered; this PR carries the corrected state and the fix so it cannot recur.

What I got wrong

proxy_token appears in proxy.env on the server and in proxy.env on each proxy — both Ansible templates. That looked exactly like the symmetry the other two secrets have.

It is not the same. The management server never reads that token from any file Ansible writes; it is registered in the management database. Rotating it changed only what the proxies present:

management:  proxy auth failed: rpc error: code = ResourceExhausted
             desc = too many failed authentication attempts     (~75/min, steady)
proxy:       http: TLS handshake error ... unknown domain "home.renner.uno"

The second follows from the first — a proxy that cannot authenticate never receives its domain configuration, so it rejects the very domains it holds valid certificates for. I initially read those as two separate problems.

The test that would have caught it

Now written into the play: a secret is safe to rotate here only if it appears in management.json.j2. turn_secret (lines 22, 26) and relay_secret (line 40) do. proxy_token does not, and nothing else in the role tree feeds it to the management server.

Templating both sides of a wire is not the same as owning both ends of a trust relationship.

Recovery

Restored the original proxy_token from the pre-rotation vault and re-applied both roles. Auth failures → 0, domain errors → 0, containers healthy.

vault_netbird_proxy_token moves back into vault.yml. It had ended up in netbird-rotatable.yml, which this play rewrites wholesale — the next run would have silently dropped it. Verified by re-running both deploys: changed=0 on all three hosts.

Also fixed

The shadow-key guard. vault.yml carried both vault_netbird_turn_secret and a bare netbird_turn_secret. The role default maps one to the other, but a group var outranks a role default, so the bare key won — rotating only the prefixed one would have changed nothing that reached the template. The guard now matches both spellings.

Diagnostic note

One of my checks was invalid and worth flagging: I tested the proxies with curl --resolve from the LAN and got HTTP 200 from both. That proved nothing — HTTP from inside is redirected to the proxy regardless of the address given. Same class as the UDM DNS interception. The hash comparison of the deployed token was the check that actually distinguished states.

Rotation ran in production. **`turn_secret` and `relay_secret` are rotated and correct.** `proxy_token` was in the same set and should never have been — rotating it locked every proxy out of the management server. Recovered; this PR carries the corrected state and the fix so it cannot recur. ## What I got wrong `proxy_token` appears in `proxy.env` on the server **and** in `proxy.env` on each proxy — both Ansible templates. That looked exactly like the symmetry the other two secrets have. It is not the same. The management server never reads that token from any file Ansible writes; it is registered in the management **database**. Rotating it changed only what the proxies present: ``` management: proxy auth failed: rpc error: code = ResourceExhausted desc = too many failed authentication attempts (~75/min, steady) proxy: http: TLS handshake error ... unknown domain "home.renner.uno" ``` The second follows from the first — a proxy that cannot authenticate never receives its domain configuration, so it rejects the very domains it holds valid certificates for. I initially read those as two separate problems. ## The test that would have caught it Now written into the play: **a secret is safe to rotate here only if it appears in `management.json.j2`.** `turn_secret` (lines 22, 26) and `relay_secret` (line 40) do. `proxy_token` does not, and nothing else in the role tree feeds it to the management server. Templating both sides of a wire is not the same as owning both ends of a trust relationship. ## Recovery Restored the original `proxy_token` from the pre-rotation vault and re-applied both roles. Auth failures → **0**, domain errors → **0**, containers healthy. `vault_netbird_proxy_token` moves back into `vault.yml`. It had ended up in `netbird-rotatable.yml`, which this play rewrites wholesale — the next run would have silently dropped it. Verified by re-running both deploys: `changed=0` on all three hosts. ## Also fixed The shadow-key guard. `vault.yml` carried **both** `vault_netbird_turn_secret` and a bare `netbird_turn_secret`. The role default maps one to the other, but a group var outranks a role default, so the bare key won — rotating only the prefixed one would have changed nothing that reached the template. The guard now matches both spellings. ## Diagnostic note One of my checks was invalid and worth flagging: I tested the proxies with `curl --resolve` from the LAN and got HTTP 200 from both. That proved nothing — HTTP from inside is redirected to the proxy regardless of the address given. Same class as the UDM DNS interception. The hash comparison of the deployed token was the check that actually distinguished states.

claude-bot added 1 commit 2026-08-26 10:56:36 +00:00

fix(ansible): stop rotating netbird_proxy_token — Ansible does not own both ends

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

6ff5cb7d7f

Rotated turn_secret and relay_secret in production; both are correct and stay
rotated. proxy_token was in the same set and should never have been: rotating
it locked every proxy out of the management server.

The misjudgement was specific. proxy_token appears in proxy.env on the server
AND in proxy.env on each proxy, both Ansible templates, which looked exactly
like the symmetry test the other two pass. It is not: the management server
never reads it from a file Ansible writes. The token is registered in the
management DATABASE, so rotating it changed only what the proxies present.

  management: proxy auth failed: rpc error: code = ResourceExhausted
              desc = too many failed authentication attempts   (~75/min, steady)
  proxy:      http: TLS handshake error ... unknown domain "home.renner.uno"

The second follows from the first — a proxy that cannot authenticate never
receives its domain configuration, so it rejects the domains it holds valid
certificates for.

Recovered by restoring the original proxy_token from the pre-rotation vault and
re-applying both roles. Auth failures went to zero and the domain errors
stopped. turn_secret and relay_secret keep their new values; both appear in
management.json.j2, so both ends really are Ansible's.

vault_netbird_proxy_token moves back into vault.yml. It was in
netbird-rotatable.yml, which this play rewrites wholesale — the next run would
have silently dropped it. Verified by re-running both deploys: changed=0.

Adds the test that would have caught this, in the play's own words: a secret is
safe to rotate here only if it appears in management.json.j2. Templating both
sides of a wire is not the same as owning both ends of a trust relationship.

Also corrects the shadow-key guard. vault.yml carried BOTH
vault_netbird_turn_secret and a bare netbird_turn_secret; the role default maps
one to the other, but a group var outranks a role default, so the bare key won
and rotating only the prefixed one would have changed nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-26 10:58:04 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 6ff5cb7d7fcb171b536bd3c173826c3a1be5648c.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Incident-recovery PR: proxy_token was mistakenly included in the NetBird secret-rotation play even though the management server never reads it from any Ansible-templated file (it's registered in the management database, not management.json). Rotating it locked out every proxy. This PR stops rotating proxy_token, restores the pre-rotation vault state, and tightens the shadow-key guard to also catch a bare netbird_turn_secret (not just the vault_-prefixed form). I verified the core claims against the checkout and they hold.

No blocking issues.

🟡 Suggestions

ansible/plays/rotate-netbird-secrets.yml:154 — the updated fail_msg lists the three keys to remove as "vault_netbird_turn_secret, netbird_turn_secret and vault_netbird_relay_secret" but omits the bare netbird_relay_secret spelling, even though the regex above it (^(vault_)?netbird_(turn_secret|relay_secret):) checks for that spelling too. If a vault ever shadows relay_secret the same way turn_secret was shadowed, the guard will correctly refuse to run, but the operator-facing message won't mention the exact key to delete. Minor, since the regex/logic itself is correct and this is just guidance text.

Tests

This is an operational Ansible playbook with no test harness in the repo (consistent with the rest of ansible/plays/). The self-verification here is behavioral: the PR body states the guard was validated live (changed=0 on all three hosts after restoring vault_netbird_proxy_token and re-running both deploys), and I independently confirmed the new grep regex matches/excludes the intended key spellings, and that management.json.j2 (ansible/roles/netbird-server/templates/management.json.j2:22,26,40) only ever reads netbird_turn_secret and netbird_relay_secret, never netbird_proxy_token — supporting the PR's central claim. No further automated test is expected or missing here.

Recommendation

Looks reasonable — no blocking issues; the suggestion above is optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `6ff5cb7d7fcb171b536bd3c173826c3a1be5648c`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Incident-recovery PR: `proxy_token` was mistakenly included in the NetBird secret-rotation play even though the management server never reads it from any Ansible-templated file (it's registered in the management database, not `management.json`). Rotating it locked out every proxy. This PR stops rotating `proxy_token`, restores the pre-rotation vault state, and tightens the shadow-key guard to also catch a bare `netbird_turn_secret` (not just the `vault_`-prefixed form). I verified the core claims against the checkout and they hold. No blocking issues. ## 🟡 Suggestions - `ansible/plays/rotate-netbird-secrets.yml:154` — the updated `fail_msg` lists the three keys to remove as "`vault_netbird_turn_secret`, `netbird_turn_secret` and `vault_netbird_relay_secret`" but omits the bare `netbird_relay_secret` spelling, even though the regex above it (`^(vault_)?netbird_(turn_secret|relay_secret):`) checks for that spelling too. If a vault ever shadows `relay_secret` the same way `turn_secret` was shadowed, the guard will correctly refuse to run, but the operator-facing message won't mention the exact key to delete. Minor, since the regex/logic itself is correct and this is just guidance text. ## Tests This is an operational Ansible playbook with no test harness in the repo (consistent with the rest of `ansible/plays/`). The self-verification here is behavioral: the PR body states the guard was validated live (`changed=0` on all three hosts after restoring `vault_netbird_proxy_token` and re-running both deploys), and I independently confirmed the new grep regex matches/excludes the intended key spellings, and that `management.json.j2` (`ansible/roles/netbird-server/templates/management.json.j2:22,26,40`) only ever reads `netbird_turn_secret` and `netbird_relay_secret`, never `netbird_proxy_token` — supporting the PR's central claim. No further automated test is expected or missing here. ## Recommendation Looks reasonable — no blocking issues; the suggestion above is optional.

k.renner merged commit e7c70ea368 into master

2026-08-26 11:04:55 +00:00

k.renner referenced this issue from a commit

2026-08-26 11:04:56 +00:00

Merge pull request 'fix(ansible): stop rotating netbird_proxy_token — Ansible does not own both ends' (#632) from netbird-rotate-migration into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#632