hermes-bot
0bc2c6b4fb
ha-token mode revoked the token it had just replaced right after storing the
new one in OpenBao. Consumers never read OpenBao: ESO copies it into the k8s
Secret on a 1h refreshInterval, and env-var consumers only see it after a
Reloader restart. On 2026-10-01 that left every consumer on a revoked token
for ~35 min: up{job="home-assistant"} was 0 from 02:01 to 02:35 UTC (401 on
the scrape -> HomeAssistantDown, TargetDown), and zone-configurator
crashlooped on "Auth invalid".
The rotator now reaps instead of revoking: after the store it lists HA's
long-lived tokens and revokes only those named "<HA_CLIENT_NAME> [" (the
rotator's own unique-name form) that are neither the stored-before token nor
the new one. The replaced token stays valid for one quarter and the next run
removes it. Same keep-the-previous pattern as gitea-token mode (59e804d9).
reap_targets is pure and unit-tested.
The HA rotation CronJobs were pinned to secret-rotator:2026-07-30 with no
automation, so a code fix would never deploy. Enrol the image in Flux image
automation (ImageRepository + ImagePolicy, newest YYYY-MM-DD tag) and mark the
three HA CronJobs only: staged, the other rotators (one runs daily) follow
once an off-cycle HA run has validated the new binary.
zone-configurator and speech-to-phrase read home-assistant-prometheus-token
as an env var with no Reloader annotation, so they would still hold the
replaced token when the next run reaps it. Add the annotation.
Closes #1084
Closes #1081
homelab
GitOps source for the Renner homelab cluster. Flux applies main to
production, so every merge is a production change. Changes go in by pull
request only; see CONTRIBUTING.md for YAML conventions.
Change rules that protect Home Assistant
Owner decision (REN-240, REN-243). Home Assistant runs the house, so it gets two rules on top of normal review.
1. Disruptive work runs only 02:00–05:00 Europe/Berlin
Planned work that can restart Home Assistant or disrupt the cluster is merged and applied only between 02:00 and 05:00 Europe/Berlin. That covers:
- node drain, node reboot, Talos upgrade
- CNI (Cilium, Multus) and cluster-wide network policy
- storage (Rook/Ceph, CSI, snapshot controller, CNPG operator)
- Flux core (
kubernetes/clusters/*/flux-system/, Flux controllers) - Authentik
- OpenBao, including secret rotation that restarts consumers
- anything in
kubernetes/apps/home-assistant/
A PR in this list states in its description that it waits for the window. Flux applies on merge, so "wait for the window" means merge in the window, not just apply. If unsure whether a change is disruptive, treat it as disruptive.
Not covered: Home Assistant image bumps by Flux image automation. They stay fully automatic by owner decision. If one breaks Home Assistant, roll it back with RUNBOOK-home-assistant-image-rollback.md. An incident rollback is not planned work; the owner decides its timing.
Also not covered, by owner decision: the Hermes ha-updates job
(kubernetes/apps/hermes/skills/ha-updates/) installs HACS and device-firmware
updates that Home Assistant offers, and restarts Home Assistant for them, on
its own inside the 02:00–05:00 window (UniFi and SMLIGHT firmware excluded;
reported only). The window is enforced by its ha.py, not only by the prompt.
2. HA-impact review before merge
Every PR that touches kubernetes/apps/home-assistant/, or makes a
cluster-wide disruptive change from the list above, needs an HA-impact
review before merge: a named reviewer states in the PR what Home
Assistant will notice (restart, lost integration, lost history, none) and
approves.
- Reviewer: Hestia, the Home Assistant maintainer.
- Until the owner hands this review to Hestia (the
hestiaHermes profile, which today only states the expected impact in its PRs): the owner does the HA-impact review.
This is in addition to the normal review and merge by Argus. Argus does not merge such a PR without the HA-impact approval.