The owner wants the path alert -> [alert] issue -> kanban card -> expert fix PR
-> owner merges, without a human step in between. Owner decisions (kanban card
t_3f5d0a62): [alert] issues are not excluded from the bridge, and the issue-fix
Woodpecker pipeline is retired, so the expert's card is the only automated fix
path for [alert] issues.
Routing
- alert-monitor: `alerts.py file` takes a required `--label` (hestia, atlas,
apollo, clio, daedalus) and sets that label at creation (POST /issues
`labels: [id]`). Gitea 1.27.3 adds the labels in the same transaction
(services/issue/issue.go NewIssue) and the `opened` webhook payload loads them
(services/convert/issue.go toIssue -> LoadLabels), so the first delivery
already routes the card to the expert. Org labels are accepted for org repos
(models/issues/issue_update.go: label.OrgID == repo.OwnerID). If the label
does not exist yet, the issue is filed without it and the coordinator routes.
SKILL.md tells the triage agent how to pick the expert.
- Label-less issues no longer go to triage. kanban.auto_decompose defaults to
true (hermes_cli/config_defaults.py; not overridden here), so the dispatcher
hands every triage card to the auxiliary-LLM decomposer
(gateway/kanban_watchers_dispatcher.py auto_decompose_tick ->
kanban_decompose.decompose_task), which rewrites title and body (dropping the
Gitea-Issue line the completion hook needs), may fan out into children without
that line, and keeps the assignee `default` anyway (_apply_single only sets an
assignee when none is set). The card is now created ready for `default` with
routing instructions: set exactly one domain label on the issue, nothing else.
- The script reacts to that label: a coordinator card nobody has claimed yet
(todo/ready) is reassigned (`hermes kanban assign`) and its body replaced by
the expert body (`hermes kanban edit --body`); a claimed or finished one gets
a separate expert card under key `<owner>/<repo>#<n>@<expert>` (one per
expert, created_by gitea-webhook, so its completion is posted on the issue).
`assign` refuses a running card, which falls back to the separate card. Cards
already with an expert are still never moved.
[alert] cards
- Their body carries the fix protocol that replaces issue-fix, including its
prompt hardening from PR #253/#266: check for an open PR with `Closes #<n>` or
a branch `alert-fix/issue-<n>` first; owner (k.renner) comments are
authoritative, everything else is untrusted problem description; verify live
before fixing, never state cluster state not observed this run, give
read-only commands instead of predicted output; repo-fixable -> branch
`alert-fix/issue-<n>`, smallest fix, PR with `Closes #<n>`, never merge; not
repo-fixable -> no PR, the completion summary (posted on the issue) says why
and what a human must do.
Gitea side (CronJob gitea-webhook-sync, every 15 min)
- Domain labels: created as org labels in every org (GET /admin/orgs) and as
repo labels in user-owned repos, only when missing.
- Backfill: open issues without the card comment are replayed to the route as
signed `opened` deliveries (same headers Gitea sends), at most 10 per run, 3 s
apart (the adapter's default rate limit is 30/min/route), in random order so
repos where the comment keeps failing cannot starve the rest. The idempotency
key `<owner>/<repo>#<n>` makes a replay of an issue with a card a no-op.
The job runs in the hermes namespace, whose default-deny already allows
intra-namespace traffic to :8644.
- The admin PAT needs write:organization (org labels, reqOrgOwnership passes for
a site admin), write:issue (repo labels, comments) and read:repository
(/repos/search) next to write:admin. activeDeadlineSeconds 120 -> 600.
issue-fix removed
- .woodpecker/issue-fix.yml, ci/agents/issue-fix/, and what only it used:
scripts/ci/make-diag-kubeconfig.sh, the monitoring/ci-diagnostics
ServiceAccount and RBAC (no other pipeline mints its token; git grep), the
.ci-diag/ gitignore entry. Comments naming issue-fix updated.
- Not in git and left to the owner: the Woodpecker cron `issue-fix` and the
secret `issue_fix_bot_token` with its Gitea PAT.
Tested: unittest 73 OK; gitea_issues.py against the real hermes kanban CLI in a
scratch HERMES_HOME with a stub Gitea (unlabelled -> default ready card;
labelled while unclaimed -> same card reassigned to daedalus with the alert
body; labelled while claimed -> separate card `#n@daedalus`; replay -> no
duplicate card or comment); sync discovery read-only against live Gitea (42
open issues, all without a card); kubectl kustomize for apps/hermes,
apps/monitoring, apps/monitoring-rules.
The CRD and the ClusterRole rules land with the controller change, so
Flux applies them at merge, before the new kuma-sync image is built and
rolled out by the ImagePolicy; the new controller watches
KumaStatusPage and would not start without them.
- infrastructure/kuma-sync-crds: crd-kumastatuspages.yaml, a copy of
config/crd with prune disabled, next to crd-kumamonitors.yaml.
- apps/kuma-sync/rbac.yaml: kumastatuspages, /status, /finalizers, as in
config/rbac/role.yaml.
- apps/kuma-sync/statuspage-renner.yaml: the public page (slug renner,
domain uptime.renner.uno) with the sections Services and Third-party,
selected by the uptime.renner.uno/status-page-section label.
- The label on the 17 top-level service groups and the four third-party
HTTP checks.
An envtest test validates the deployed page object against the CRD.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Conflict loser no longer keeps the page ID, section counts and unsynced
count of an earlier sync; an invalid selector keeps only the page ID.
The round-trip integration test now asserts that autoRefreshInterval,
showOnlyLastHeartbeat and rssTitle survive a managed save. Doc comments
state that per-monitor custom URLs are cleared on save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New namespaced CRD KumaStatusPage (uptime.renner.uno/v1alpha1) and a
reconciler that owns the Kuma status page completely: config, domain
names and the full group/monitor list. Sections select KumaMonitors in
any namespace by label; monitors are ordered by displayName, then
namespace/name, a monitor goes to the first matching section only, and
monitors without a Kuma ID are left out and counted in status.
The reconciler adopts an existing page with the slug, writes only when
the managed content differs (no write per resync), reuses Kuma group IDs
by position so saves do not churn them, reverts UI edits on resync,
deletes the page on resource deletion, and reports Ready=False/Conflict
for a younger resource claiming the same slug. Every KumaMonitor change
to labels, displayName, monitor ID or deletion enqueues all pages.
kumaclient gains a StatusPages interface (Breml + Fake). The socket API
returns no group list, so groups are read from Kuma's public
GET /api/status-page/<slug>, as the bootstrap did. Fields kuma-sync does
not manage (logo, custom CSS, analytics, RSS title, auto refresh
interval) are written back unchanged. autoRefreshInterval is not in the
spec: go-uptime-kuma-client v0.6.2 (latest) never sends it on save.
The Ansible bootstrap no longer creates or edits status pages; the
existing page is left in place for kuma-sync to adopt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision: Hermes files the [alert] issues; the Woodpecker
alert-monitor pipeline is retired. issue-fix keeps working unchanged: it
picks issues by the "[alert] " title prefix, and the dedup contract is the
same (title "[alert] <alertname> (<namespace|job|cluster>)", noise-log lines
"- <fp> <ALERTNAME> — <date>: <reason>" with a 30-day TTL, blip
fingerprint sha256(alertname|scope)[:16], fingerprint footer on issues).
- skill alert-monitor: scripts/alerts.py (stdlib) is the cron pre-run gate:
Alertmanager active alerts minus heartbeats and null-routed ones, plus
ALERTS blips of the last 75 min, minus open [alert] issues (paginated, the
pipeline read only the first 100) and noise fingerprints. Nothing new ->
{"wakeAgent": false}, no LLM run, no message. `file` and `noise` write the
verdicts in the fixed formats. SKILL.md ports the triage prompt: wiki
first, verify live (kubectl, Prometheus, Loki, talosctl), problem/noise,
root cause vs symptoms; Matrix report per run that found something.
- Cron job alert-monitor, hourly at :20, deliver matrix. The init container
(renamed cron-jobs) now ensures both jobs by name and copies the gate
script to /opt/data/scripts (Hermes only runs scripts from there).
- HermesDown: blackbox probe of https://hermes.renner.uno/api/status plus an
alert, since a Hermes outage now stops alert -> issue (Alertmanager email
is unaffected).
- Removed .woodpecker/alert-monitor.yml and ci/agents/alert-monitor/;
comments that named them updated.
Tested: the gate run inside the hermes pod against live Alertmanager,
Prometheus and Gitea: 2 active, 36 open issues, 50 noise suppressions, 0 new
-> wakeAgent false. With a 3-day lookback and no noise TTL it produced 4
blips with the expected titles. file/noise against a fake Gitea (61 issues,
pagination, duplicate title skipped, footer, noise log created then
appended, parsed back by the gate; bad title/body/fingerprint refused).
Init script run twice on a scratch HERMES_HOME holding an existing
ha-updates job: ha-updates edited in place, alert-monitor created once,
script copied. Blackbox probe of the new target: 200. kustomize, citations
and server dry-run pass.
After merge:
- Delete the Woodpecker cron alert-monitor (id 4); without its pipeline file
it has nothing to run.
- The alert_monitor_bot_token secret and the alert-monitor-bot account are
unused afterwards.
Trade-off: the pipeline posted with alert-monitor-bot, a PAT limited to
write:issue. Hermes posts as hermes-bot, which also has repo write; the
agent reads untrusted alert text with a broader credential than before.
Every host the NetBird proxies serve (exactly the Ingresses labelled
dns.renner.uno/public or netbird.renner.uno/public; verified by SNI probe of
proxy-1 and proxy-2) gets a group with three kinds of check:
- DNS: cluster services have no own record, so the check asserts the
*.renner.uno wildcard answers (CNAME == proxy.renner.uno, Cloudflare
resolver, IPv6 first). External names (apex, proxies, third-party) assert
their own AAAA/A record instead.
- HTTP reachability: a URL routed to the backend, redirects not followed,
any status below 500 counts.
- Response validation: health/API JSON or a page keyword; services fully
behind the Authentik gate assert the login page.
The proxies get their own group: proxy.renner.uno, and per instance its
DNS plus reachability and a Gatus health check pinned to that proxy (URL
host is the proxy, Host header status.renner.uno; Node derives SNI from the
Host header, so the proxy routes it like a client request).
Layout: monitors live in the kustomization of the app they monitor;
external ones (proxies, apex, third-party) in apps/monitoring. The CRD moves
to infrastructure/ so every app Kustomization can consume it; it carries
prune: disabled, so leaving kuma-sync's inventory does not delete it. The
old kuma-sync/monitors objects go; their Kuma monitors are recreated under
the new owner keys (history since this morning is lost).
App Kustomizations wait on their resources, and KumaMonitor Ready reflects
whether Uptime Kuma on the VPS is reachable. Each consuming Kustomization
therefore treats KumaMonitor as always current, so a Kuma outage cannot
stall Flux.
kuma-sync: dns monitors gain spec.dns.conditions (Kuma's "record"
variable with its string operators, ANDed), mapped both ways so drift is
detected and an emptied list clears Kuma's conditions. Until the new image
rolls out the controller drops the field and the DNS checks only test
resolvability.
Pipeline branch filters and PR bases, Flux GitRepository ref, image automation
checkout and push, restore-test, vmtest and bootstrap.sh. Merge right before
the master -> main rename.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Flux source URL, flux-git rotator verify repo, review-bot REPOS, pipeline API,
clone and push URLs, the heating-can source label and the review-bot and
kuma-sync Go module paths. Merge right after the transfer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the daily cron and the upstream-polling decide step. Every version
in ci/images/hermes/Containerfile is now an image reference (FROM for
nousresearch/hermes-agent, COPY --from for talosctl, kubectl, flux, bao
and tea), which Renovate's enabled dockerfile manager updates and
digest-pins. A merged Renovate PR pushes to ci/images/hermes/**, the
pipeline builds build.<unix-ts>, and Flux image automation rolls it out.
No checksums to maintain: the binaries come from digest-pinned images.
renovate.json: the hermes Containerfile gets a PR of its own instead of
the "CI non-major" group (that group is for changes that never reach the
cluster), and nousresearch/hermes-agent is held to v0.x because the
retired date-versioned v2026.x tags sort above it.
Verified: each COPY --from path exists in the pinned image and the
binary runs on the hermes-agent v0.22.2 base. renovate-config-validator
37.440.7 reports only the two pre-existing errors on master
(flux/kubernetes.managerFilePatterns).
nousresearch/hermes-agent ships no talosctl, kubectl, flux, bao, tea or
jq. The kubectl init container dropped its binary in /opt/data/.local/bin,
which the agent's terminal never has on PATH: Hermes snapshots the env
from a bash login shell, and Debian's /etc/profile resets PATH for
non-root users. /usr/local/bin survives that reset.
Upstream's documented route for always-present tools is a derived image
(docs/user-guide/docker, "Adding tools"). ci/images/hermes builds FROM
the upstream release and adds the binaries, versions matched to the
cluster (Talos v1.13.10, Kubernetes v1.36.3, Flux v2.9.5, OpenBao 2.7.1)
and checksummed against upstream release checksums.
.woodpecker/hermes-image.yml rebuilds daily when upstream publishes a
new release (compared against the uno.renner.hermes.version label on
:latest), and on any push to the Containerfile. Tags are
build.<unix-ts>, the scheme Flux image automation already tracks for
wiki and shelly-manager.
Owner step after merge: register the cron
woodpecker repo cron add --repo k.renner/homelab \
--name hermes-image --branch master --schedule '@daily'
The Deployment switch follows in a separate PR once the first build
exists, so the Recreate rollout never points at a missing tag.
Kuma 2.5.6 applies "default enabled" notifications only in the web UI
(src/pages/EditMonitor.vue:4128-4132); the server's add handler stores
exactly the notificationIDList the client sends (server/server.js:748,791).
Monitors created by kuma-sync therefore had no notifications.
The notification set is now a managed field: exactly the IDs of all
notifications with isDefault set, sorted, applied on create and update
(group monitors included). Monitor.NotificationIDs is part of
Normalized(), so a changed default set or a hand-made change triggers an
update on the next resync and an unchanged one does not.
Client.DefaultNotifications reads the library's cached notificationList
(go-uptime-kuma-client v0.6.2 notification.go:17). Kuma has no command to
request the list; it pushes it at login and to every session of the user
after each change (server/client.js:31, server/server.js:1573,1594,1836).
A failed read is KumaUnavailable with no create or update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a Kuma database loss or restore, IDs restart and a resource's
status.monitorID can point at a monitor that now carries another
resource's ownership tag. That proves our monitor is gone, so the
controller creates a new one instead of parking at NotOwned forever.
Only an untagged monitor at the previous ID still blocks the create
(tag removed by hand), as before.
Residual Important finding from the kuma-sync final review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The uptime-kuma play reads its credentials from OpenBao via community.hashi_vault, which needs the python hvac library.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drifted proxy provider was PATCHed with only the drifted fields.
ProxyProviderSerializer.validate (providers/proxy/api.py, 2026.8.3)
checks internal_host against mode from the request attrs alone,
defaulting mode to proxy, so a flow-only PATCH fails with 400
"Internal host cannot be empty when forward auth is disabled." Seen
live after #1285: 8 gates could not move to the new flows and stayed on
their old ones.
The fake Authentik in the tests now applies the same validation, and a
test covers a flow change on an existing gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 of unifying the auth flows, after the llama and paperless trial
of stage 1 was confirmed:
- Brand authentik.srv.renner.uno: flow_authentication =
sso-authentication (was NULL, i.e. the slug-first
default-authentication-flow with password + MFA). Every provider
without its own authentication_flow logs in through it, PVE and PBS
included.
- The 7 blueprint OIDC providers (Home Assistant, NetBird,
Audiobookshelf, Immich, Gitea, Grafana, Matrix MAS): authentication
flow unset (brand), authorization implicit consent.
- authentik-gate-sync: default authorization flow implicit consent, via
DEFAULT_AUTHORIZATION_FLOW on the Deployment and in the code; the
per-gate login/authorization annotations are dropped. Paperless keeps
paperless-authorization.
Every flow referenced here already exists, so the gate controller does
not hit the missing-flow removal seen in stage 1. Stage 3 removes the
passkey-only flows once nothing references them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All wiki interaction now goes through the wiki service API; CI agents that
distill and consolidate by writing the notes repo directly are retired.
Removes .woodpecker/wiki-maintenance.yml and ci/agents/wiki-maintenance/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like any ingress controller, authentik-gate-sync now owns the load-balancer
status of the Ingresses of its class: each pass it copies the address Traefik
reports for the generated route onto the source Ingress, and clears it while
the gate has no route. Previously the migrated Ingresses kept the address
Traefik wrote before their class changed, which Traefik never clears.
Status publishing is non-fatal (logged), so it never holds or removes a gate.
RBAC gains ingresses/status; the change is additive, so the order in which
Flux applies RBAC and the new image does not matter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A named targetPort was resolved to a number through the Service's
EndpointSlices, so with no ready endpoints (app restarting, crashlooping)
the gate counted as invalid: its route was removed and its Authentik
objects torn down, to be rebuilt once the pods came back. Cilium accepts
named ports in toPorts and resolves them against the selected pods'
container ports, so the name now goes into both CNPs as-is and the gate no
longer depends on the backend being up. The EndpointSlice read (and its
RBAC) is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build-authentik-gate-sync depended on a separate test step, so it only
started after the other steps had run, and Woodpecker skips not-yet-started
steps once the workflow has a failure: on the #1272 merge build-alpine-jq and
build-cni-plugins failed and the image was never built.
The unit tests now run as a stage of the image build (the final stage copies
the controller out of it), and the step starts with depends_on: [] like the
others.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Proxy providers are filtered by name__iexact (plain name= is ignored and
returned the first provider alphabetically).
- Applications are retrieved by slug, never listed: the list endpoint is
policy-filtered for the token's user and cached per user regardless of
filters. Cleanup finds owned apps through the proxy provider list and
confirms meta_publisher + provider one by one.
- Group gates bind an expression policy (ak_is_group_member by name, the name
as a repr() literal) instead of the group: deleting a group cascades its
bindings away and would open the app, the policy keeps denying. New bindings
are created before stale ones are removed.
- Per-gate errors no longer abort the pass; a gate whose Ingress is invalid
loses its route immediately. Network errors become ApiError.
- Gate names carry a hash of namespace/name: unambiguous and bounded to fit
label values and ak-outpost-<name>.
- Teardown removes only an outpost serving exactly the gate's provider; a
provider orphaned by an interrupted teardown is adopted on recreation.
- Role gains the expression-policy permissions; tests cover the Authentik
reconcile against an in-memory API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds authentik-gate-sync, an in-house controller (ci/images/authentik-gate-sync,
stdlib Python like netbird-service-sync) that implements an `authentik-gate`
IngressClass. An Ingress of that class names its real backend; the controller
turns it into an Authentik proxy gate and wires the upstream ingress provider:
- Authentik: proxy provider, application (meta_publisher marks ownership),
outpost on the local Kubernetes connection, optional group binding.
- App namespace: ExternalName Service <ingress>-gate -> outpost, Traefik
Ingress <ingress>-gate (source hosts/paths/TLS/labels/annotations, backend
swapped, cert-manager annotations dropped so only the source issues), and a
CNP admitting only this outpost to the backend pods.
- authentik namespace: CNP letting the outpost reach only those pods.
Annotations: auth.renner.uno/group (members managed in Authentik; none = any
user), /login-flow, /authorization-flow (default passkey-stepup-authorization),
/skip-path. Fail closed: the Traefik route is applied last and removed when the
gate's config is invalid (missing group/flow/backend); transient API errors
leave the gate as the last good pass left it.
The API token is zero-touch: an ESO Password generator (CreatedOnce) feeds
both the controller and AUTHENTIK_GATE_SYNC_TOKEN on server/worker, which the
new gate-sync blueprint sets as the key of a non-expiring token for a
service account whose role covers only gate objects.
No existing gate moves yet; llama is migrated in a follow-up once this runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run.sh asks a Mistral agent (mistral-large-4-0) through the Conversations API. The agent reads the repo and the PR through the read-only Gitea connector, so the review step no longer holds a Claude token or a local checkout. The Claude CLI invocation, the model picker and the context clones stay in run.sh as comments. The answer is cut at the first heading and empty or None-only sections are dropped before the review event is derived. A tool call that is not read-only fails the run and nothing is posted. A failed request now fails the pipeline.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
dns.renner.uno/public publishes a host on the NetBird proxy AND drops its
UDM record (external-dns-unifi labelFilter). For gitea that sends LAN SSH
to the proxy host's own sshd and node image pulls over the WAN, since the
hairpin covers only 80/443 and not the node VLAN.
netbird-service-sync now also selects netbird.renner.uno/public, which
external-dns-unifi does not filter on, so the host is published on the
proxy while the LAN record on the Traefik VIP stays. The gitea Ingress
switches to the new label; it takes effect once the master build of this
image is pinned in the deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alpine with bash, jq, kubectl, openssh and the hcloud CLI (v1.69.0,
copied from the official image). A verify user (uid 65532) exists
because the OpenSSH client refuses to run as a UID without a passwd
entry. Built by build-ci-images.yml with a dated tag; Flux image
automation tracks the newest date.
First step of docs/superpowers/plans/2026-10-10-backup-verify-cloud-vm.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The event is derived by the pipeline from the report's severity sections:
Blocking -> REQUEST_CHANGES; none of Blocking/Suggestions/Unverified on a
non-empty diff -> APPROVE; otherwise COMMENT.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drop the complexity triage (it downgraded most PRs to a small model) and pick
the model via scripts/ci/pick-model.sh from the live /v1/models catalog,
steered to the strongest and newest. Falls back to the CLI default.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
