www.multi-circuit-boards.eu (third-party Apache) occasionally resets a
single TLS handshake with ECONNRESET. Kuma 2.5.6 opens a fresh
connection for every check, so two such resets in a row were enough
for DOWN with retryInterval 60 / maxRetries 1: 14 false episodes since
2026-10-20 21:51 UTC. A 30-min probe from the cluster (559 requests at
3s) saw 557x 200 and 2 handshake resets, and got 200 during Kuma's
DOWN window.
With retryInterval 20 (CRD minimum) and maxRetries 3, DOWN now takes 4
failures in a row over at least 60s, so a real outage longer than
about 1 min still alerts. The other third-party monitors are left
unchanged.
Refs #1407
- predates() also guards approvals: an approval older than the card the PR
line names now (e.g. a follow-up whose expert rebutted without a push)
no longer completes that card on replay.
- predates() parses created_at as epoch or RFC 3339 and fails loudly when a
card has none (hermes kanban show --json gives epoch seconds, v0.22.2,
checked against the CLI), instead of silently never skipping.
- Closing one PR does not complete a card another open PR (stacked stage)
still names.
- Follow-up cards tell the worker to read every review and comment since the
previous card: only the newest finding is quoted, older ones that predate
the card are not added (deliberate, now pinned by a test).
review-bot 1802 on #1417 (blocking): marker dedupe was per card, but the PR's
Hermes-Card line moves (follow-up card, coordinator hand-off, gitea_pr.py
adopt), so a replay re-sent findings already handled on the earlier card to
the new one. A finding older than the card the line names now is skipped
(review submitted_at / comment created_at vs the card's created_at); the new
card's worker reads the PR itself.
Also from the review:
- PR merged or closed while its card is in review completes the card
(hook event pull_request_only, route event pull_request, closed hermes-bot
PRs of the replay window are replayed); a rebuttal without push otherwise
left the card in review forever once the owner merged.
- 404 on the PR (system hook fires for repos hermes-bot cannot read) is a
skip, not an error line per review.
- One route-script run per PR at a time (flock): a delivery and its replay
can overlap.
- find_review drops dismissed reviews like the replay does.
- Replay checks the run budget before listing; CronJob comment updated.
- Skill: stun-tls-probe.py runs via python3 (ConfigMap mode 0644); rebuttal
wording says the card waits for the owner.
Experts closed their card with kanban_complete as soon as the PR was open, and
nothing listened to Gitea reviews afterwards (#1412/#1414 REQUEST_CHANGES and
the review-bot suggestions on #1411/#1415 sat unanswered).
Owner decision: the card stays in review until the Gitea review is through; no
follow-up cards for a card that is still open.
- Herkunft: expert PRs carry `Hermes-Card: <card> (<profile>)`. The new helper
homelab-gitea-pr/scripts/gitea_pr.py sets it deterministically (open/adopt)
and refuses a second PR for the same branch or `Closes #N` (find/open).
The card id comes from --card, $HERMES_KANBAN_TASK or the scratch workspace
name: the terminal tool does not pass HERMES_KANBAN_TASK to commands.
- Events: the system hook adds pull_request_review and pull_request_comment
(Gitea 1.27.3 routers/api/v1/utils/hook.go; HasEvent maps the approved/
rejected/comment review types onto pull_request_review). Headers per
modules/webhook/type.go Event(): pull_request_approved/_rejected/_comment
for reviews, issue_comment for PR comments; the route lists exactly those.
- Routing in gitea_issues.py (no LLM, [SILENT], HMAC by the gateway as before):
the review payload carries no review id or commit, so the script reads the
reviews API and ignores reviews of an older head.
review-bot APPROVED on the head + card in review -> hermes kanban complete.
review-bot REQUEST_CHANGES/COMMENT, owner reviews with text, owner PR
comments -> card in review: hermes kanban reopen-review (back to the
implementer from review_requested, review quoted as untrusted text); card
open: comment; card done/missing: follow-up card for the same expert
(key <owner>/<repo>!<n>@after-<card>, needed because Hermes has no CLI to
reopen a done card) and the PR's line moves to it; PR without a line:
coordinator card (key ...@review). A marker per review/comment id makes
every replay a no-op.
`hermes kanban request-changes` is not usable here: it needs an active
reviewer run claimed from review, which only exists with review dispatch.
- kanban.review_dispatch=false on default (Hermes docs: "Set false for
human-only review boards"); otherwise every card in review would spawn an
sdlc-review LLM run next to review-bot.
- Replay step in gitea-webhook-sync: review-bot's latest review and owner
reviews of each open hermes-bot PR's head plus owner PR comments (48 h) are
re-delivered every 15 min. review-bot can approve before the expert calls
kanban_request_review; without the replay that card would stay in review.
- homelab-gitea-pr moves from the coordinator's PVC into git (experts never
had it): review-loop section, one-PR-per-cause rule, the helper. SOUL.md and
the bridge's card text say: PR cards end with kanban_request_review.
Addresses review-bot review 1792 on a1b52f5b:
- K8sGatewayProbesAbsent now fires per record (A, AAAA), so one broken
module no longer hides behind the other's series. New promtool case
(AAAA-only input) and mutation k8s-gateway-absent-aaaa-dropped.
- K8sGatewayIngressAnswerWrong description names the other cause: the
auth.renner.uno Ingress (authentik) being gone.
- The Traefik lb-ipam-ips annotation points back at the blackbox modules
that match the same VIP.
Folds the good parts of #1412 into this PR and addresses both review-bot
reviews (#1412 review 1789, #1414 review 1791):
- Everything after the OpenBao store is best-effort. Decode, reconnect,
listing and revokes run in reap_superseded(); reap_summary() turns any
error into "reap skipped" plus a warning, so rotate_ha_token returns Ok.
A failed run would start a second pod (backoffLimit: 1) that keeps the
token just stored and its successor and reaps the token consumers still
hold until ESO syncs, i.e. the #1081 outage. Two unit tests cover it.
- Warn when the replaced token is not named "<client_name> [<6 hex>]":
no later run reaps it, so it needs a manual revoke (from #1412).
- Staged image rollout (from #1412): only the three HA CronJobs carry the
flux-system:secret-rotator marker; the other 17 rotators stay on
2026-07-30 until a manual HA run validates the new binary, then a
follow-up adds the marker to them. rotation/kustomization.yaml records
the exception.
- Merged current main (a6f37e2b, 1691a0ac): the Reloader annotations for
zone-configurator and speech-to-phrase are already there via #1409.
review-bot (review 1787) noted the old explanatory comment was dropped without replacement. The range stays open-ended: the homelab rule is to never pin a Helm chart version, Traefik follows current upstream.
review-bot: /users/{u}/orgs requires the user and organization scope
categories (Gitea 1.27.3 api.go, HasScope needs all), and the sync's PAT
has write:admin,write:organization,write:issue,read:repository, no
read:user. /admin/users/{u}/orgs needs only the admin category and lists
private orgs for a site admin.
Also: drop paged()'s dead /repos/search unwrapping, test the team-mode
fallback of team_writes_issues, and say in SKILL.md that direct-collaborator
and user repos still get cards via the system hook.
From 2026-10-08 to 2026-10-16 k8s-gateway ran the chart's default Corefile
(zone example.com, forward . /etc/resolv.conf). Pods stayed Ready, every
renner.uno lookup was forwarded to the UDM and still answered (public names as
CNAME proxy.renner.uno), and nothing alerted. probes-dns.yaml only asks the UDM.
Two blackbox DNS modules ask the k8s-gateway Service (by name, not IP) for
auth.renner.uno A over IPv4 (the path CoreDNS forwards on, 10.96.0.11) and
AAAA over IPv6, and fail unless every answer RR is the Traefik VIP
(10.2.99.86 / fd99::86). A forwarded answer — CNAME or any other address —
fails. CoreDNS is bypassed on purpose: its renner.uno block serves stale for 1h.
K8sGatewayIngressAnswerWrong (critical) fires when the 10m success average is
under 0.8 for 5m: about 7m after a total failure starts, and also when one of
the two replicas is broken (average ~0.5), where a `for` on the raw value would
keep resetting. One or two failed probes stay silent. K8sGatewayProbesAbsent is
the dead-man's switch.
Verified: the module config was run with blackbox_exporter 0.29.0 against the
live cluster resolver (which relays k8s-gateway's answer): A and AAAA succeed;
a k8s-gateway-forwarded name (ha.renner.uno, 10.2.99.87) and a wildcard name
(CNAME proxy.renner.uno) fail. promtool-rules.sh with promtool 3.15.0: 9 suites
green, 41 mutations caught (4 new).
Closes#1290
ha-token mode revoked the outgoing LLAT seconds after storing the new one in
OpenBao. Consumers read the k8s Secret, which ESO syncs every 1h, so on
2026-10-01 the ServiceMonitor scrape got 401 from 02:00 to ~02:36Z
(HomeAssistantDown, TargetDown) and zone-configurator crashlooped (#1081,
#1082, #1084).
The reap now keeps the token just stored and the one it replaced, and
revokes only rotator-minted tokens (exact "<client_name> [6 hex]") other
than those two, the same keep-two rule gitea-token mode got in 59e804d9.
The replaced token is revoked by the following quarterly run.
The binary only reaches the CronJobs through a new image, and every
rotator was hand-pinned to secret-rotator:2026-07-30. Enrol the image in
Flux image automation (dated-tag policy, as for ci-backup-verify) and mark
all 20 rotator image lines, so they move together to the newest built tag.
Source between 2026-07-30 and this commit differs only by the Rust 1.91 ->
1.99 build image (68c09786).
Closes#1081
The sync reached only konairius/homelab ("repos: 1 with issues writable by
hermes-bot") although hermes-bot has write on all 10 org repos through the
org team Bots (includes_all_repositories, repo.issues write).
Cause: GET /repos/{o}/{r}/collaborators/{user}/permission (Gitea 1.27.3
GetRepoPermissions) returns permission.AccessMode, the repo-wide mode from
the collaborator/access level. Rights from a unit-based team land only in
the per-unit modes and are not part of that field, so it said "none" for
the 9 repos where hermes-bot is not a direct collaborator (checked live:
homelab "write", shelly-manager and ical-proxy "none", while
/orgs/konairius/repos reports push: true for all 10).
Scope is now derived from org membership: orgs of hermes-bot
(/users/hermes-bot/orgs) -> teams granting repo.issues write that it is a
member of -> all org repos (includes_all_repositories) or the team's repos,
non-archived with issues on. User-owned repos are out of scope, so labels
are org labels only and the repo-label branch is gone.
Tests: team scope, read-only team skipped, selected-repo team, user repos
out of scope, failed team lookups, org-labels-only. Live read-only probe
with hermes-bot's token: 10 repos, label target konairius.
Chart 41.0.0 renamed `logs.access` to `accessLog` and its values schema
rejects the old root `logs` key; that is why 399f101b capped the range
below 41.0.0. Moving `logs.access.enabled` to `accessLog.enabled` makes
our values valid on the current line, so the cap goes.
Verified with helm v4.3.1 against the published charts, values taken
from this HelmRelease, --kube-version 1.36.3:
- 42.0.0 with the old values fails exactly as in June
("additional properties 'logs' not allowed").
- 42.1.0 with these values renders the same manifests as the running
40.3.0, apart from chart/version labels and the image
(traefik v3.7.4 -> v3.7.15). The access-log args are unchanged
(--accesslog=true, fields defaultmode keep, headers drop), so the
CrowdSec agents parsing Traefik logs see the same format.
- Chart CRDs: same 25 CRDs; additive schema changes and controller-gen
0.17.1 -> 0.21.0 annotations.
Other breaking notes for 41/42 do not apply here: providers.file.content
is not set; 42.0 requires Kubernetes >= 1.30 (cluster runs 1.36.3);
the Hub secret RBAC and transparency-log witness changes only apply
with hub.token, which is not set.
The range is now open-ended (>=42.0.0), per the house rule not to pin
chart versions; note the wiki's floating-helm-chart-constraint-risk
recommends tilde ranges instead, so the owner should confirm the shape.
Refs #426
Regenerated with `flux install --export
--components-extra=image-reflector-controller,image-automation-controller`
from the v2.9.6 CLI (release tarball, sha256 checked against
flux_2.9.6_checksums.txt); the same component set as before.
Controllers: source v1.9.5 -> v1.9.6, kustomize v1.9.5 -> v1.9.6, helm
v1.6.4 -> v1.6.5. notification, image-reflector and image-automation
are unchanged. No CRD, RBAC or NetworkPolicy changes: the diff is the
three images and their app.kubernetes.io/version labels.
`flux check --pre` (v2.9.6 CLI) passes on Kubernetes 1.36.3.
Flux core: merge only 02:00-05:00 Europe/Berlin.
Refs #426
review-bot on 5de84e94: the guard sat after up to 20000 characters of untrusted
issue text. render() now takes a head block placed right after the title line;
Gitea-Issue: stays the last line.
Also checked (Gitea release/v1.27 routers/api/v1/repo/collaborators.go,
GetRepoPermissions; the API docs only list 200/403/404): for a site-admin caller
it answers 200 with the access mode (none for a non-collaborator), 404 only for an
unknown user, so repos hermes-bot cannot see are filtered, not failed.
review-bot on b6179a41:
- Expert card bodies open with "This card is for the <expert> expert. If you are
not <expert>, do no work on it": a coordinator that claims the card between the
body edit and the reassign completes it instead of running the alert protocol
next to the separate @expert card.
- issue_repos skips (and names) a repo whose hermes-bot permission lookup fails
instead of aborting; the other repos still get labels and backfill, the Job
still fails.
review-bot on de080442:
- The site-admin PAT moves from apps/hermes/gitea-admin (readable by the agent's
`hermes` role, homelab/data/apps/hermes/*) to apps/gitea-webhook-sync/admin.
Only ESO reads it (eso role, existing ClusterSecretStore); `bao token
capabilities` with the hermes role: deny on the new path. ExternalSecret and
K8s Secret are now gitea-webhook-sync-admin, mounted only by the CronJob.
- Hand-over writes the expert body before reassigning. If the coordinator claims
the card in between, its routing body is restored and the expert gets the
separate @expert card; if the edit fails, nothing is reassigned. Tests for both.
- [alert] cards: a comment is the owner's only by its API user.login (k.renner,
the login issue-fix trusted and the one commenting on homelab issues), never by
a name or quote in the text.
- Labels and backfill only in repos where hermes-bot has issue write (per
/collaborators/hermes-bot/permission), org labels only for orgs owning such a
repo; gitea_admin/_cargo-index (hermes-bot: read) no longer gets labels.
- Backfill starts no replay after RUN_BUDGET_SECONDS (900); activeDeadlineSeconds
1200 covers that plus one 60 s delivery; a test pins the relation.
Verified live: Gitea 1.27.3 swagger says IssueLabelsOption.labels takes IDs or
names, so the coordinator's {"labels": ["<expert>"]} works; hermes-bot has
write on konairius/homelab.
The owner wants the path alert -> [alert] issue -> kanban card -> expert fix PR
-> owner merges, without a human step in between. Owner decisions (kanban card
t_3f5d0a62): [alert] issues are not excluded from the bridge, and the issue-fix
Woodpecker pipeline is retired, so the expert's card is the only automated fix
path for [alert] issues.
Routing
- alert-monitor: `alerts.py file` takes a required `--label` (hestia, atlas,
apollo, clio, daedalus) and sets that label at creation (POST /issues
`labels: [id]`). Gitea 1.27.3 adds the labels in the same transaction
(services/issue/issue.go NewIssue) and the `opened` webhook payload loads them
(services/convert/issue.go toIssue -> LoadLabels), so the first delivery
already routes the card to the expert. Org labels are accepted for org repos
(models/issues/issue_update.go: label.OrgID == repo.OwnerID). If the label
does not exist yet, the issue is filed without it and the coordinator routes.
SKILL.md tells the triage agent how to pick the expert.
- Label-less issues no longer go to triage. kanban.auto_decompose defaults to
true (hermes_cli/config_defaults.py; not overridden here), so the dispatcher
hands every triage card to the auxiliary-LLM decomposer
(gateway/kanban_watchers_dispatcher.py auto_decompose_tick ->
kanban_decompose.decompose_task), which rewrites title and body (dropping the
Gitea-Issue line the completion hook needs), may fan out into children without
that line, and keeps the assignee `default` anyway (_apply_single only sets an
assignee when none is set). The card is now created ready for `default` with
routing instructions: set exactly one domain label on the issue, nothing else.
- The script reacts to that label: a coordinator card nobody has claimed yet
(todo/ready) is reassigned (`hermes kanban assign`) and its body replaced by
the expert body (`hermes kanban edit --body`); a claimed or finished one gets
a separate expert card under key `<owner>/<repo>#<n>@<expert>` (one per
expert, created_by gitea-webhook, so its completion is posted on the issue).
`assign` refuses a running card, which falls back to the separate card. Cards
already with an expert are still never moved.
[alert] cards
- Their body carries the fix protocol that replaces issue-fix, including its
prompt hardening from PR #253/#266: check for an open PR with `Closes #<n>` or
a branch `alert-fix/issue-<n>` first; owner (k.renner) comments are
authoritative, everything else is untrusted problem description; verify live
before fixing, never state cluster state not observed this run, give
read-only commands instead of predicted output; repo-fixable -> branch
`alert-fix/issue-<n>`, smallest fix, PR with `Closes #<n>`, never merge; not
repo-fixable -> no PR, the completion summary (posted on the issue) says why
and what a human must do.
Gitea side (CronJob gitea-webhook-sync, every 15 min)
- Domain labels: created as org labels in every org (GET /admin/orgs) and as
repo labels in user-owned repos, only when missing.
- Backfill: open issues without the card comment are replayed to the route as
signed `opened` deliveries (same headers Gitea sends), at most 10 per run, 3 s
apart (the adapter's default rate limit is 30/min/route), in random order so
repos where the comment keeps failing cannot starve the rest. The idempotency
key `<owner>/<repo>#<n>` makes a replay of an issue with a card a no-op.
The job runs in the hermes namespace, whose default-deny already allows
intra-namespace traffic to :8644.
- The admin PAT needs write:organization (org labels, reqOrgOwnership passes for
a site admin), write:issue (repo labels, comments) and read:repository
(/repos/search) next to write:admin. activeDeadlineSeconds 120 -> 600.
issue-fix removed
- .woodpecker/issue-fix.yml, ci/agents/issue-fix/, and what only it used:
scripts/ci/make-diag-kubeconfig.sh, the monitoring/ci-diagnostics
ServiceAccount and RBAC (no other pipeline mints its token; git grep), the
.ci-diag/ gitignore entry. Comments naming issue-fix updated.
- Not in git and left to the owner: the Woodpecker cron `issue-fix` and the
secret `issue_fix_bot_token` with its Gitea PAT.
Tested: unittest 73 OK; gitea_issues.py against the real hermes kanban CLI in a
scratch HERMES_HOME with a stub Gitea (unlabelled -> default ready card;
labelled while unclaimed -> same card reassigned to daedalus with the alert
body; labelled while claimed -> separate card `#n@daedalus`; replay -> no
duplicate card or comment); sync discovery read-only against live Gitea (42
open issues, all without a card); kubectl kustomize for apps/hermes,
apps/monitoring, apps/monitoring-rules.
Every open issue on gitea.renner.uno lands on the Hermes kanban board without a
manual step (owner decision: all issues, no opt-in label). Exactly one label out
of hestia/atlas/apollo/clio/daedalus assigns the card to that expert; anything
else goes to the coordinator (default) in triage.
Inbound, event-driven through upstream Hermes features only (webhook platform,
route script, kanban CLI; nothing forked or patched):
- A Gitea system webhook (events issues_only + issue_label) posts to
hermes.hermes.svc.cluster.local:8644/webhooks/gitea-issues. Gitea signs with
X-Hub-Signature-256 and sends X-GitHub-Event: issues, which the Hermes webhook
adapter validates as GitHub-style HMAC.
- Route gitea-issues runs skills/gitea-issues/scripts/gitea_issues.py as its
route script. The script creates the card itself with
`hermes kanban create --idempotency-key <owner>/<repo>#<number>
--created-by gitea-webhook` and answers [SILENT], so no agent run and no LLM
call happens. Edits and label changes reuse the existing card (no duplicate,
no reassignment). The issue text is quoted into the card body as untrusted
input; the card ends with a `Gitea-Issue: <url>` line.
- It then comments the card id on the issue once (marker comment, so retries
and later events do not repeat it).
Back channel: a kanban_task_completed shell hook (config `hooks:` on default and
all five experts) runs the same script in `completed` mode and posts the
completion summary on the issue. Dispatcher-spawned workers run with
--accept-hooks; the gateway (dashboard completions) gets an allowlist entry in
each profile's shell-hooks-allowlist.json. The issue is not closed
automatically: a finished card is not proof the issue is resolved (the work may
sit in an unmerged PR), so closing stays with the owner or an explicit ask.
Gitea side, declarative: CronJob gitea-webhook-sync (hermes ns, every 15 min,
hermes image, scripts/sync_webhook.py) keeps the system hook in place through
the admin API. Gitea 1.27.3 facts it relies on (routers/api/v1/utils/hook.go):
POST /admin/hooks creates a *default* (template) hook unless
config.is_system_webhook is "true", and PATCH ignores config.secret. So the
hook is named hermes-kanban-<hash of url, events, secret>; when no active hook
has the desired name, a new one is created first and older hermes-kanban* hooks
are deleted afterwards. Other system hooks (review-bot) are never touched.
Secrets (OpenBao, apps/hermes only, bootstrap commands in
gitea-webhook-externalsecret.yaml):
- apps/hermes/gitea-webhook#secret: HMAC secret, read by the gateway (written
to /opt/data/.env as WEBHOOK_SECRET by default-env, which also sets
WEBHOOK_ENABLED only when the secret exists) and by the sync job.
- apps/hermes/gitea-admin#api-token: site-admin PAT (write:admin; Gitea has no
narrower scope for system hooks), mounted only into the sync job.
The Deployment carries a Reloader annotation for hermes-gitea-webhook, so the
first sync of the secret (and a rotation) restarts the gateway with it.
Network: Service port webhook 8644, CNP hermes-allow-gitea (ingress from the
gitea pod on 8644), egress rule in gitea's default-deny, and
hermes.hermes.svc.cluster.local in GITEA__security__ALLOWED_HOST_LIST.
Tested offline: tests/test_gitea_issues.py (routing, card shape, marker
spoofing, comment dedup, completion, sync create/no-op/replace). Against the
real Hermes code in a scratch HERMES_HOME with a stub Gitea API: WebhookAdapter
accepted a Gitea-style signed POST (200, script), rejected a bad signature
(401), ignored a pull_request event; repeated issue events produced one card;
the init-container logic set the route, hooks and allowlist (`hooks list`:
allowed); kanban_db.complete_task after register_from_config fired the hook and
posted the summary.
Registers https://home.renner.uno/api/mcp as mcp_servers.homeassistant via
an init container (same pattern as the other config set steps), so the
entry is reapplied on every start instead of living only on the PVC.
Auth reuses Hermes's own HA admin token (apps/hermes/homeassistant, already
synced to hermes-homeassistant for ha-updates), exposed as HA_TOKEN and
referenced as ${HA_TOKEN} in the header; Hermes interpolates it at connect
time, so the token never lands in config.yaml.
trust: untrusted makes every tool without readOnlyHint (HassTurnOn, scripts,
media, vacuum, ...) require approval; read-only tools (GetLiveContext,
calendar/todo reads) run freely. This keeps the standing "no agent write
calls to the HA API" decision enforced per call instead of silently lifted.
TURN/TLS never reached LiveKit. A TLS connection to
turn.matrix.renner.uno without ALPN, as TURN clients open it, got
LiveKit's HTTP "OK" from the HTTP router; a STUN Binding request on it got
no answer.
Traefik v3.7 (pkg/server/router/tcp/router.go, ServeTCP) matches HTTPS
routers before TCP-TLS routers: an HTTPS router with a specific Host takes
precedence over an IngressRouteTCP with the same HostSNI. The matrix-rtc
Ingress created exactly that HTTPS router for the turn host.
The turn host moves to its own Ingress bound to the `web` entrypoint only,
so netbird-service-sync still publishes it (it reads Ingress hosts) but no
HTTPS router claims the SNI on websecure. Its certificate becomes a
standalone Certificate (livekit-turn-tls), since the Ingress no longer
carries a tls section.
The LAPI has been in CrashLoopBackOff since the restart #1394 triggered
(10:08Z). Its own watcher login fails:
tls: failed to verify certificate: x509: certificate has expired or is
not yet valid: current time 2026-10-20T10:13:44Z is after
2026-10-20T06:24:37Z
Rolling back to the previous ReplicaSet fails the same way, so any LAPI
restart since this morning would have hit it.
Cause: the chart's CA Certificate sets no duration, so cert-manager's 90d
default applies, the same lifetime as the leaves (tls.certManager.duration
90d, renewBefore 30d). All of them renew together. At 2026-09-20 06:24:37
the leaves were re-signed by the CA issued 2026-07-22 06:24:37 while the
CA itself rotated to a new key. That earlier CA expired today at 06:24:37,
so every leaf now chains to an expired CA. The LAPI pod running since
2026-10-01 kept working with the certs already loaded, until the restart.
The same skew is what made Traefik reject the LAPI cert (500) since
2026-09-20.
Fix, no chart fork: our own CA in tls-ca.yaml (self-signed Issuer, 10y CA
Certificate renewed a year before expiry, CA Issuer), handed to the chart
via tls.certManager.issuerRef. The chart then stops rendering its own
CA/issuers. cert-manager re-issues every leaf because the Secret's issuer
annotation no longer matches the Certificate's issuerRef
(SecretIssuerAnnotationsMismatch). Leaves renew every 60d under a CA that
lives 10y, so the leaves and the CA no longer renew together.
Reloader restarts LAPI (crowdsec-lapi-tls, crowdsec-agent-tls) and agents
(crowdsec-agent-tls) when the certs change; both read them at start only.
The ServersTransport already trusts crowdsec-lapi-tls's ca.crt (#1394), so
it follows the new CA.
LiveKit crashlooped after #1393:
could not parse "tcp://10.107.186.157:7880" as uint value from
environment variable "LIVEKIT_PORT" for flag port
The Service named livekit makes the kubelet inject Docker-style link
variables (LIVEKIT_PORT, LIVEKIT_SERVICE_HOST, ...) into every pod in the
namespace. LiveKit reads LIVEKIT_* as its own config; lk-jwt-service reads
LIVEKIT_* too. enableServiceLinks: false on both pods removes them.
crowdsec.renner.uno has answered 500 on every path since 2026-09-20.
Traefik logs, every request:
tls: failed to verify certificate: x509: certificate signed by unknown
authority (possibly because of "x509: ECDSA verification failure" while
trying to verify candidate authority certificate "crowdsec-ca")
Cause: the chart's CA (crowdsec-ca, no duration set, so cert-manager's 90d
default) and its leaves (tls.certManager.duration 90d, renewBefore 30d) were
created together and so renew at the same instant. cert-manager logs show
crowdsec-ca, crowdsec-lapi and crowdsec-agent all triggered "Renewing" at
2026-09-20T06:24:37Z. The CA came back with a new ECDSA key, while the
leaves issued in the same second were signed by the previous one. The
ServersTransport trusted crowdsec-root-secret, so it holds the new CA and
rejects the LAPI cert. Same subject, different key, hence "ECDSA
verification failure".
Fix: point the ServersTransport at the ca.crt of crowdsec-lapi-tls. For a
CA issuer, cert-manager stores the signing CA there, so it matches the
served cert whichever key the root holds. Traefik reads ca.crt from a
rootCAs secret (Traefik v3.7 docs).
Reloader also restarts the LAPI when crowdsec-lapi-tls changes. The LAPI
reads its cert at start; without a restart a renewal would update ca.crt
while the old leaf is still served, and the mismatch would come back.
Impact while broken: the edge firewall bouncers on proxy-1/proxy-2 got no
decisions from this endpoint.
Element X reported MISSING_MATRIX_RTC_TRANSPORT: matrix.renner.uno had no
MatrixRTC backend (no SFU, /rtc/transports 404, msc4143 off).
Adds the backend from the Element Call self-hosting guide and the
lk-jwt-service README:
- LiveKit SFU (rtc.yaml), signalling at wss://rtc.matrix.renner.uno.
Media: ICE straight to the pod IPs (UDP 7882 mux, TCP 7881), which the
LAN reaches through the BGP-routed pod CIDR, dual-stack. Clients that
cannot reach the pod (mobile data, NetBird-only) use the embedded TURN
server over TLS on turn.matrix.renner.uno:443. That rides the existing
public path: NetBird proxy TLS passthrough -> Traefik, where an
IngressRouteTCP terminates TLS (turn.external_tls) and hands plain TCP to
LiveKit. The proxy only does TLS on :443, so TURN/TLS is the one transport
that works from the internet without opening router ports.
- lk-jwt-service as a Synapse appservice (MSC4502 membership look-ups,
MSC4512 proxy of /rtc/livekit to it), plus its legacy client endpoints
under rtc.matrix.renner.uno/livekit/jwt: Element Call 0.28 / Element Web
1.12.32 still use livekit_service_url.
- The legacy endpoints verify OpenID tokens at
<server>/_matrix/federation/v1/openid/userinfo, found through
/.well-known/matrix/server (fallback :8448). Synapse therefore serves the
server well-known and the `openid` listener resource, and the Ingress
routes exactly those two paths. No federation resource is served and
federation_domain_whitelist stays empty; federation remains off.
- Synapse: msc4143/4222/4502/4512, matrix_rtc transport (url +
livekit_service_url), rtc_foci in the client well-known for older
embedded Element Call builds, delayed events (24h) and the guide's rate
limits for key sharing and membership heartbeats.
- Secrets: one Password generator (LiveKit API secret, appservice tokens),
generated once in-cluster like the former matrix-bridge tokens; Synapse,
LiveKit and lk-jwt-service read the same Secret, so nothing can drift and
nothing needs seeding in OpenBao.
- CNPs: world -> LiveKit 7881/tcp + 7882/udp and LiveKit -> world UDP
(ICE checks/media); lk-jwt-service -> Traefik 8443 (matrix.renner.uno and
rtc.matrix.renner.uno resolve to the Traefik VIP in-cluster).
- embedded TURN denies private peers by default; the relay targets the
SFU's own pod IP, so 10.0.0.0/8 and fc00::/7 are allowed (LiveKit already
limits embedded-TURN relays to its own media ports).
- Kuma: DNS + reachability for rtc. and turn.matrix.renner.uno.
Every host the NetBird proxies serve (exactly the Ingresses labelled
dns.renner.uno/public or netbird.renner.uno/public; verified by SNI probe of
proxy-1 and proxy-2) gets a group with three kinds of check:
- DNS: cluster services have no own record, so the check asserts the
*.renner.uno wildcard answers (CNAME == proxy.renner.uno, Cloudflare
resolver, IPv6 first). External names (apex, proxies, third-party) assert
their own AAAA/A record instead.
- HTTP reachability: a URL routed to the backend, redirects not followed,
any status below 500 counts.
- Response validation: health/API JSON or a page keyword; services fully
behind the Authentik gate assert the login page.
The proxies get their own group: proxy.renner.uno, and per instance its
DNS plus reachability and a Gatus health check pinned to that proxy (URL
host is the proxy, Host header status.renner.uno; Node derives SNI from the
Host header, so the proxy routes it like a client request).
Layout: monitors live in the kustomization of the app they monitor;
external ones (proxies, apex, third-party) in apps/monitoring. The CRD moves
to infrastructure/ so every app Kustomization can consume it; it carries
prune: disabled, so leaving kuma-sync's inventory does not delete it. The
old kuma-sync/monitors objects go; their Kuma monitors are recreated under
the new owner keys (history since this morning is lost).
App Kustomizations wait on their resources, and KumaMonitor Ready reflects
whether Uptime Kuma on the VPS is reachable. Each consuming Kustomization
therefore treats KumaMonitor as always current, so a Kuma outage cannot
stall Flux.
kuma-sync: dns monitors gain spec.dns.conditions (Kuma's "record"
variable with its string operators, ANDed), mapped both ways so drift is
detected and an emptied list clears Kuma's conditions. Until the new image
rolls out the controller drops the field and the DNS checks only test
resolvability.
#1383 + konairius/Wiki#14 put the wiki's read API behind an OIDC session
or an API key. External agents only have the wiki-index.renner.uno
BasicAuth credentials and got 401 since.
konairius/Wiki (feat/basic-auth-machines) checks `Authorization: Basic`
in the service against the htpasswd lines Traefik uses. This mounts the
existing Secret wiki-api-basic-auth (OpenBao apps/ai/wiki-index-basic-auth,
already the wiki's own) read-only at /var/run/secrets/wiki-basic-auth.
Auth__Basic__UsersPath points at the directory, because the service reads
every key there, as Traefik does. A rotated Secret now also rolls the pod
(reloader annotation).
Order does not matter: the current image ignores the setting, and the new
image without it authenticates nobody over BasicAuth. Traefik's middleware
on wiki-index stays.
The dashboard console's /api/pty and /api/events WebSockets close
together every ~12-23 s (Traefik durations 18.9/21.8/22.7/22.8 s on
2026-10-19 11:14-11:19Z), the browser reports code 1006 and shows
"Please wait while the conversation loads…". The TUI process and
/api/status stay healthy meanwhile. The only ~20 s server timer found is
uvicorn's WS keepalive (dashboard.ws_ping_interval, default 20 s).
Diagnostic change: if socket lifetimes move to ~60 s the keepalive is
implicated; if they stay at ~20 s it is ruled out. Set via init container
like dashboard.trusted_proxies since there is no env equivalent.
The dashboard logged out exactly ~60 min after every login
(dashboard-auth.log: session_rejected reason=no_cookie at login+60m).
Hermes' self-hosted OIDC provider defaults to "openid profile email" and
the Authentik hermes-provider mapped only those scopes. Since authentik
2024.2 a refresh token is issued only when the client requests
offline_access and the provider has the offline_access scope mapping, so
Hermes never got one and could not rotate the session when the ID-token
cookie expired.
- authentik blueprint: add the managed scope-offline_access mapping to
hermes-provider (refresh_token grant was already enabled).
- hermes: HERMES_DASHBOARD_OIDC_SCOPES adds offline_access.
Stage 3 of 3. Merge only after konairius/Wiki feat/oidc-login is running
(Deployment image newer than build.1790322994 and `wiki-79948db4c6` gone):
the current image has no login of its own, so on plain Traefik its UI
would be open to the LAN.
- Ingress `wiki`: ingressClassName traefik instead of authentik-gate.
authentik-gate-sync's sweep then deletes what it generated for the gate
(Ingress/Service/CNP wiki-gate in ai, CNP gate-ai-wiki-c9ba6b in
authentik, the ai-wiki-c9ba6b outpost, application and provider). One
proxy layer fewer: Traefik -> wiki.
- Deployment: the Auth__ReverseProxy__TrustedProxies entries go. They were
a hand copy of the Talos podSubnets, and the OIDC image no longer reads
them.
- Middleware wiki-api-strip-identity goes. No header names a user any
more; wiki-index.renner.uno keeps BasicAuth.
- Comments in ai-ns.yaml and the Authentik blueprint follow.
The `wiki` CNP already admits Traefik on 8080, and the ai namespace egress
already reaches Traefik, so auth.renner.uno (OIDC discovery, JWKS, token)
needs no new rule.
Stage 1 of 3 for moving the wiki off proxy-header identity
(konairius/Wiki feat/oidc-login).
The wiki took the username from X-Authentik-Username for any peer in the
pod CIDR, so any pod that may reach :8080 (Hermes, Gitea, Traefik, all of
ai and monitoring) could claim a human's name. Authentik's proxy provider
sends no signed identity header, so the wiki now signs humans in itself
with OpenID Connect and takes the username from the verified ID token.
- Blueprint: OAuth2 provider "Wiki" (public client `wiki`, PKCE,
authorization_code, RS256 self-signed key, openid+profile) and
application `wiki` (issuer https://auth.renner.uno/application/o/wiki/),
redirect URI https://wiki.renner.uno/signin-oidc. No policy binding,
the same audience as today's gate, which has no group annotation.
- Deployment: Auth__Oidc__Authority and Auth__Oidc__ClientId. The running
image ignores them; the OIDC image refuses to start without them, which
is why this lands first.
Order: this PR, then konairius/Wiki feat/oidc-login (Flux image
automation rolls it; it works behind the existing gate, since the gate
session makes the OIDC round trip silent), then the stacked PR that
moves wiki.renner.uno off the authentik-gate class.
The hermes ServiceAccount cannot list nodes, so the InternalIP lookup
via kubectl get nodes failed (Forbidden). talosctl accepts node names
through the talos.default endpoint: talosctl -n k8s-2 get members and
-n k8s-1,k8s-2,k8s-3 version both succeed from the hermes pod.
registry-bot is the konairius registry identity: write:package PAT,
konairius org secret registry_bot_token, rotated by
openbao/rotation/registry-bot.yaml. That rotation syncs the secret for
the push, tag, release and manual events, so the manual trigger is
`event: manual` instead of a deployment target.
registry_bot_token is a konairius org secret. Woodpecker resolves org
secrets through the repo's org link, so this pipeline gets it only once
homelab has moved to konairius/homelab
(docs/superpowers/plans/2026-10-18-homelab-konairius-main.md, cut-over 1).
Drops the daily cron and the upstream-polling decide step. Every version
in ci/images/hermes/Containerfile is now an image reference (FROM for
nousresearch/hermes-agent, COPY --from for talosctl, kubectl, flux, bao
and tea), which Renovate's enabled dockerfile manager updates and
digest-pins. A merged Renovate PR pushes to ci/images/hermes/**, the
pipeline builds build.<unix-ts>, and Flux image automation rolls it out.
No checksums to maintain: the binaries come from digest-pinned images.
renovate.json: the hermes Containerfile gets a PR of its own instead of
the "CI non-major" group (that group is for changes that never reach the
cluster), and nousresearch/hermes-agent is held to v0.x because the
retired date-versioned v2026.x tags sort above it.
Verified: each COPY --from path exists in the pinned image and the
binary runs on the hermes-agent v0.22.2 base. renovate-config-validator
37.440.7 reports only the two pre-existing errors on master
(flux/kubernetes.managerFilePatterns).
nousresearch/hermes-agent ships no talosctl, kubectl, flux, bao, tea or
jq. The kubectl init container dropped its binary in /opt/data/.local/bin,
which the agent's terminal never has on PATH: Hermes snapshots the env
from a bash login shell, and Debian's /etc/profile resets PATH for
non-root users. /usr/local/bin survives that reset.
Upstream's documented route for always-present tools is a derived image
(docs/user-guide/docker, "Adding tools"). ci/images/hermes builds FROM
the upstream release and adds the binaries, versions matched to the
cluster (Talos v1.13.10, Kubernetes v1.36.3, Flux v2.9.5, OpenBao 2.7.1)
and checksummed against upstream release checksums.
.woodpecker/hermes-image.yml rebuilds daily when upstream publishes a
new release (compared against the uno.renner.hermes.version label on
:latest), and on any push to the Containerfile. Tags are
build.<unix-ts>, the scheme Flux image automation already tracks for
wiki and shelly-manager.
Owner step after merge: register the cron
woodpecker repo cron add --repo k.renner/homelab \
--name hermes-image --branch master --schedule '@daily'
The Deployment switch follows in a separate PR once the first build
exists, so the Recreate rollout never points at a missing tag.
Switches apps/hermes to gitea.renner.uno/gitea_admin/hermes, which
carries talosctl, kubectl, flux, bao, tea and jq in /usr/local/bin. The
image-automation policy now tracks its build.<unix-ts> tags instead of
upstream nousresearch/hermes-agent; .woodpecker/hermes-image.yml follows
upstream releases.
Drops the kubectl init container and its emptyDir: the binary sat in
/opt/data/.local/bin, which the agent's terminal does not have on PATH.
hermes-bot