docs(mosquitto): propose enrolling broker credentials in Class-A rotation #453
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#453
Reference in New Issue
Block a user
Documentation only — nothing implemented, no manifests changed. Written up first because the last three attempts at this each hit a different wall.
The gap
kubernetes/infrastructure/openbao/rotation/holds 14 rotators. Five are Class-A ("self-generated random secret"): authentik, crowdsec, paperless, shelly-manager, woodpecker. A broker password is exactly that shape — and the mosquitto credentials are the only Class-A-shaped secrets not enrolled.They're minted once by seven bootstrap Jobs and then never change: no schedule, no compromise response, no path at all. They also exist only in the Secret objects, so they're outside the OpenBao snapshot and unrecoverable if lost.
Why the obvious fixes don't work
Recorded because each was tried, and none of the reasons are visible from the code:
Passwordgeneratorspec.templateimmutability. Renaming works but recurs on every future edit.The common thread: each treats the credential as something to place, when the requirement is one that can be replaced.
Two mechanics that shape the design
rm -fthenmosquitto_passwd -c …for all 8 users, from env backed by the Secrets). So a Secret change is inert until the broker restarts — and there's no Reloader annotation on it today, unlike the consumers of the existing Class-A rotators.Who holds what — the part that decides cost
Not visible from the manifests:
healthzigbee2mqttevccsml2mqtt/open3eansible-vaulthomeassistantshelliesadminNote
sml2mqttandopen3ekeep the same password in two unlinked places — the Kubernetes Secret andansible-vault— synchronised by hand. That duplication, not any client limitation, is what makes rotation look impossible today. I previously claimed these "can't rotate"; that was wrong, and this documents why.Proposal
Follow
rotation/paperless.yamlexactly: generate random bytes → merge-patch OpenBao → ESO syncs the Secret → restart the consumer. Per-user rotators, an ExternalSecret each, a Reloader annotation on the broker, and the edge host reading OpenBao instead of duplicating in ansible-vault.Rotate one user at a time, so each rotation affects exactly one client rather than all eight.
Phased
health→zigbee2mqtt→evcc→ edge-host pair →shellies→homeassistant, starting with the two that need no external coordination and ending with the highest blast radius. Bootstrap Jobs get deleted incrementally as their users migrate — which also sidesteps the immutability problem entirely, since they're never patched.Open questions, listed rather than guessed
secret-rotatorrandom mode fit unchanged?RUNBOOK-cold-rebuild.md, so broker bring-up isn't blocked waiting for a rotator.Unchanged by this
Secret names, key names, every
secretKeyRefinmosquitto-deployment.yaml, the ACL model, and theevccExternalSecret which already matches the target.🤖 Generated with Claude Code
🤖 Automated review — model
claude-fable-5(complexity: trivial), commit4b5918e87e904e92b2977dd2c3e32e0e7b638ad2.Summary
Adds a documentation-only proposal (
kubernetes/apps/home-assistant/PROPOSAL-mosquitto-credential-rotation.md) to enrol the 8 mosquitto broker credentials in the existing Class-A rotation pattern. No manifests change. I verified the document's factual claims against the checkout and they hold up almost entirely; there are no blocking issues.Claims verified against the repo:
rotation/contains 15 CronJobs, one of which (claude-oauth-secret-sync) is a sync rather than a rotator, so 14 is correct. The Class-A set (authentik, crowdsec-bouncer-key, paperless, shelly-manager, woodpecker) matches the file headers.kubernetes/apps/home-assistant/mosquitto-deployment.yaml:68-76(rm -fthenmosquitto_passwdfor all 8 users, from env backed by the Secrets).kubernetes/apps/paperless/paperless.yaml:58, which has one.homeassistant-passwordon one Secret,passwordon the rest) — confirmed atmosquitto-deployment.yaml:150-196.jqand POSTing withcurl— confirmed inmosquitto-bootstrap.yaml.mosquitto-evcc-credentials-externalsecret.yamlextractsapps/home-assistant/mosquitto-evcc-credentials, matching the proposed path convention.vault_edge_sml2mqtt_mqtt_password/vault_edge_open3e_mqtt_passwordexist inansible/inventory/group_vars/edge_host/vars.ymlandansible/roles/edge-host/defaults/main.yml.rotation/home-assistant.yaml(comment at line 9 and thews://…/api/websocketendpoints).kubernetes/infrastructure/openbao/snapshot-cronjob.yaml) andkubernetes/RUNBOOK-cold-rebuild.mdboth exist as referenced.🟡 Suggestions
maxSurge: 1/maxUnavailable: 0(mosquitto-deployment.yaml:17-20), so during the broker restart two broker pods with different passwd files sit behind the LB. A client reconnecting mid-rollout can land on the old pod and fail auth until the rollout completes. Still seconds-to-a-minute and self-healing, but the failure mode ("client retries against the old pod") is worth a sentence, since it would look alarming in logs during the firsthealth/zigbee2mqttrotation.refreshInterval: 1h; the paperless rotator relies on a 10m refresh. Rotator-writes-OpenBao → Secret-updates → Reloader-restarts is delayed by up to one refresh interval, which stretches the rotation completion time (not the outage window, since broker and client move together — but it matters for "did the rotation land" verification and for the cold-rebuild ordering question already listed).🟢 Nits
mosquitto-bootstrap.yamlis 508 lines, not 518 (Problem section). The SA/Role/RoleBinding are inside that file, so "518 lines plus a ServiceAccount/Role/RoleBinding" double-counts slightly either way.Tests
Not applicable — documentation-only, no manifests or code changed, nothing for CI to validate beyond the file existing. The document itself is the artifact, and its checkable claims were verified above.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
All four corrected in
6e47ef43, each verified against the repo first — a proposal whose only value is precision should not take corrections on trust either.🟡 Rollout window: two brokers, two passwd files
Confirmed —
maxUnavailable: 0/maxSurge: 1. This is the best point in the review, because it is a failure mode that would show up during the first and safest rotation (health/zigbee2mqtt) and look like the design was broken when it is working as intended. Now documented in advance: mid-rollout two broker pods with different passwd files sit behind the LB, a reconnecting client can land on the old one and fail auth, and it self-heals when the rollout completes.🟡 ESO refresh latency
Added to the open questions.
evccusesrefreshInterval: 1h, so a rotation takes up to an hour to land. Worth being exact about what that delays: not the outage window — broker and client still move together — but how long until the rotation is complete, which matters for verifying one landed and for the cold-rebuild ordering. It also raises a real design choice I had not considered: whether these ExternalSecrets want a shorter interval, or whether the rotator should trigger the sync directly rather than waiting for a poll.🟡 "all 8 at once" → 7
Correct.
evccis already ESO-extract-backed, so a generator adoption would hit the 7 Job-minted credentials only. Fixed.Note the other "all eight" in the document (§Proposed design, point 5) is deliberate and stays: there really are 8 broker users, and the point there is that rotating one at a time affects one client rather than all of them.
🟢 508 vs 518 lines
Correct, and the double-count too — the SA/Role/RoleBinding live inside that same file, so "518 lines plus a ServiceAccount/Role/RoleBinding" was wrong twice. Now "508 lines — the ServiceAccount, Role and RoleBinding are inside that same file".
Separately: I said I would wait for this review and then did not read it until prompted. That is the second time in this session. The review was submitted at 13:42Z and sat unread.
🤖 Automated review — model
CLI default(complexity: complex), commit6e47ef4368b0bf089050ae54b449edc0144b69c6.Summary
Documentation-only proposal (
PROPOSAL-mosquitto-credential-rotation.md, new file, no manifests touched) for enrolling the 8 mosquitto broker credentials in Class-A secret rotation. I checked its factual claims against the actual manifests (mosquitto-bootstrap.yaml,mosquitto-deployment.yaml, therotation/directory,secret-rotator/src/main.rs, theevccExternalSecret, and the ansible edge-host vars) and nearly everything checks out precisely — line counts, user lists, key names, rolling-update strategy, the evcc ExternalSecret pattern, the shelly-manager rotator's scope, and the ansible-vault variable names all match the repo exactly. One factual claim doesn't hold up. No blocking issues since nothing is implemented yet.🟡 Suggestions
PROPOSAL-mosquitto-credential-rotation.md:9— "kubernetes/infrastructure/openbao/rotation/holds 14 rotators" doesn't match the directory. It contains 13 files (kustomization.yamllists 13 resources), and one of them (home-assistant.yaml) defines 3 separate rotator CronJobs (prometheus-token-secret-rotate,config-sync-token-secret-rotate,shelly-manager-llat-secret-rotate), for 15 CronJobs total. Under either way of counting (files or individual rotator CronJobs) the number isn't 14. Given the document's stated ethos is verifying claims rather than asserting them ("verified... not assumed", "recorded because each was tried and rejected on evidence"), worth correcting this one before merge so the count doesn't undermine that credibility. The Class-A list itself (authentik, crowdsec, paperless, shelly-manager, woodpecker) is correct — I confirmed all five via their# Class-Aheader comments.🟢 Nits
None beyond the above.
Tests
N/A — documentation-only change, nothing to test. The doc appropriately scopes itself as "nothing implemented" and defers verification questions (whether
secret-rotatorrandom mode fits unchanged, shelly-manager device push, HA WebSocket config-entry update) to the open-questions section rather than asserting them.Recommendation
Looks reasonable — no blocking issues; the "14 rotators" count in the Suggestions section is worth a quick fix, but nothing here blocks merging a proposal doc.
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commitff526f98c282b298a331ac898990e4cd7530e075.Summary
This PR adds a well-researched design proposal for enrolling mosquitto broker credentials in automated Class-A rotation. The proposal identifies a concrete gap (8 broker users, only one already rotated), documents why three alternative approaches failed (backed by testing), and proposes a phased implementation following the established Class-A pattern. It's documentation-only — nothing implemented, no manifests changed — and the technical analysis is sound.
🟢 Nits
evccis already ESO-extract-backed and so unaffected)." — Minor wording: this could say "extracts" instead of "extract-backed" for clarity, but the meaning is clear in context.Tests
Not applicable — this is a design proposal with no code changes.
Recommendation
Looks reasonable — no blocking issues; nit above is optional.
Clean round — no blocking, no 🟡.
🟢 "extract-backed" wording
Leaving as-is, deliberately. The term is doing real work: it distinguishes an ExternalSecret using
dataFrom.extractagainst a stored OpenBao entry from one driven by a generator — which is exactly the distinction that sentence turns on, since the generator overwrites and the extract does not. "extracts" reads more naturally but loses that contrast.More to the point: pushing a commit to change a synonym would trigger another full review round for zero substantive gain. Not a good trade.
State of this PR
Documentation only, four rounds of review, every factual claim now verified against the repo rather than asserted:
mosquitto-bootstrap.yaml— 508 lines, SA/Role/RoleBinding inside that same filemaxUnavailable: 0/maxSurge: 1, so two brokers with different passwd files coexist mid-rollevccatrefreshInterval: 1h, now an open question rather than an unstated assumptionReady to merge or discard as a proposal; nothing in it is implemented and no manifest is touched.