main
434
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  claude-botandClaude Opus 5.5
|
37c83ed6c2 |
feat(kuma-sync): declare the public status page as a KumaStatusPage
New namespaced CRD KumaStatusPage (uptime.renner.uno/v1alpha1) and a reconciler that owns the Kuma status page completely: config, domain names and the full group/monitor list. Sections select KumaMonitors in any namespace by label; monitors are ordered by displayName, then namespace/name, a monitor goes to the first matching section only, and monitors without a Kuma ID are left out and counted in status. The reconciler adopts an existing page with the slug, writes only when the managed content differs (no write per resync), reuses Kuma group IDs by position so saves do not churn them, reverts UI edits on resync, deletes the page on resource deletion, and reports Ready=False/Conflict for a younger resource claiming the same slug. Every KumaMonitor change to labels, displayName, monitor ID or deletion enqueues all pages. kumaclient gains a StatusPages interface (Breml + Fake). The socket API returns no group list, so groups are read from Kuma's public GET /api/status-page/<slug>, as the bootstrap did. Fields kuma-sync does not manage (logo, custom CSS, analytics, RSS title, auto refresh interval) are written back unchanged. autoRefreshInterval is not in the spec: go-uptime-kuma-client v0.6.2 (latest) never sends it on save. The Ansible bootstrap no longer creates or edits status pages; the existing page is left in place for kuma-sync to adopt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
ab1f004b48 |
fix(uptime-kuma): mount the bootstrap script directory, not the file
A single-file bind mount pins the inode present at container start. ansible.builtin.copy replaces the file by rename, so the running container kept executing the previous kuma-setup.js: the #1358 deploy ran the old admin-only script (host md5 8d77..., container 55a2...) and created no notification, status page or metrics key. The script now lives in {{ uptime_kuma_dir }}/setup/, mounted read-only at /app/setup; the compose change recreates the container once. The old file path is removed. Verified on a local 2.5.6 container: a file replaced by rename in the mounted directory is visible inside, and the script run from /app/setup resolves socket.io-client and converges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
7e1b46108a |
fix(uptime-kuma): fail when a list event is missing after login
Kuma 2.5.6 afterLogin (server/server.js:1839-1848) sends the notification and API-key lists inside Promise.allSettled and acks the login regardless. A missing notificationList was read as empty and would create a second default "email" notification (double mails); a missing apiKeyList would add a key without removing the old one. The script now fails the run instead. Verified against a local 2.5.6: fresh run creates everything, second run reports no changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
67aadbc0f9 |
feat(uptime-kuma): bootstrap settings, email notification, status page and metrics key
kuma-setup.js now logs in after the admin setup and converges on every run: - settings: Trust Proxy on, Primary Base URL https://kuma.renner.uno. Read-modify-write; unset keys get the same defaults the Kuma settings page applies before saving (a raw getSettings payload has nscd unset, which makes setSettings stop nscd). - notification "email" (SMTP, default enabled, applied to existing monitors): created or updated in place by name, never duplicated. Credentials from OpenBao homelab/infrastructure/uptime-kuma/smtp; recipient is the Alertmanager receiver admin@renner.uno; 465 means implicit TLS, else STARTTLS. - status page "renner" / "Renner" with custom domain uptime.renner.uno. Only title and domain are enforced; saveStatusPage replaces all groups, so the current publicGroupList from /api/status-page/<slug> is sent back unchanged. Steady state does not call saveStatusPage at all. - metrics API key "prometheus": Ansible checks the key stored at homelab/infrastructure/uptime-kuma-metrics against /metrics; if it is missing or rejected the script deletes all "prometheus" keys, creates a new one and hands it over via a 0600 file in the data dir, which Ansible writes to OpenBao (vault_kv2_write, community.hashi_vault 7.1.0 in the CI image) and then removes. The key never reaches stdout/stderr or the Ansible log. The script is restructured into small async steps with per-emit timeouts. The uptime-kuma-vps policy gains read on the smtp path and read/create/update on the metrics path (must be applied by hand). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
64a5281bd0 |
fix(uptime-kuma): dual-stack egress network for Kuma checks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
b136362fa8 |
fix(uptime-kuma): guard the setup fail check against an undefined rc
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
8ef8486b2e |
fix(uptime-kuma): start setup after Kuma's info event
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
811171646e |
fix(uptime-kuma): surface setup script output on failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
cddf555ba6 |
fix(uptime-kuma): keep the setup marker inside the data volume
The marker next to data/ survived a wiped or replaced DB, so the first up -d exposed a fresh unclaimed Kuma on /setup. Inside data/ it lives and dies with the DB; it is still written only after a successful setup run and checked before the first up -d. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
67ce7daafa |
feat(uptime-kuma): Uptime Kuma on the NetBird VPS
Role, play and pipeline deploy Uptime Kuma 2.5.6 behind the VPS Traefik. The admin user is bootstrapped over socket.io (needSetup/setup) with the route disabled, so /setup is never exposed; a .setup-done marker keeps the route enabled on later runs. Credentials come from OpenBao homelab/infrastructure/uptime-kuma via community.hashi_vault; the uptime-kuma-vps policy grants read on that one path. Only kuma.renner.uno is routed; uptime.renner.uno stays unrouted until its DNS moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
atlas-botandClaude Opus 5.5
|
4daee5fcea |
chore(edge): retire sml2mqtt for good (REN-802)
The SML power-meter reader was retired by owner decision on 2026-09-19: grid data comes from the E380 on the CAN bus (wiki sml-to-e380-grid-meter-migration). Its reader has been unplugged since then. - register: edge-1 loses write sml2mqtt/# (live, no broker restart) - bridge templates: no sml2mqtt topic, no legacy sml2mqtt bridge - remove role power-meter-sml, play install-power-meter.yml, inventory group power_meter and the vault indirection - edge-mqtt-bridge play: pre_tasks stop and remove the leftover compose project on edge-1 on the next pipeline run (idempotent) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
atlas-botandClaude Opus 5.5
|
1b3c73dbe4 |
feat(edge): bridge credential pushed by mqtt-account-controller (REN-1307)
edge-1's bridge password moves out of ansible-vault. A small receiver, edge-mqtt-cred, speaks the controller's esp32 API over HTTPS with mTLS (only CN mqtt-account-controller from the MQTT CA) and writes one bridge `edge-1` for open3e and sml2mqtt. Until the first push the two vault-password bridges stay in bridge.d/legacy.conf, so nothing changes on the cluster side yet. A Woodpecker pipeline runs the role-only play on merge, so no manual play run is needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
adbfd3bfc6 |
feat(netbird): retire the reverse proxy on the management host
The public edge is proxy-1/proxy-2; proxy.renner.uno has not pointed at netbird.renner.uno since the netcup fleet took over. The proxy that kept running on the management host has served nothing since the controller moved every service to TLS passthrough (2026-09-07): passthrough routes live on the proxy's own :443 listener, while the host's Traefik forwards HostSNI(*) to its :8443 main listener, which answers "unknown domain". Its watchdog therefore failed every probe round and ran `docker restart` every ~3 minutes since 2026-09-07 12:53 (last healthy round). Each restart registered a fresh proxy peer and pushed a network-map update to every peer in the mesh — ~10 stale proxy-* identities per ingress peer and a map update every 1-2 minutes. - netbird-server: drop the proxy service, its Traefik TCP passthrough, proxy.env and the proxy vars; `compose up --remove-orphans` removes the container, and one-off tasks stop/remove both watchdogs and delete the proxy volumes, config and state. - install-netbird-server.yml no longer applies netbird-proxy-watchdog; its pipeline no longer triggers on that role. - Comments that described the management host as a cluster member now describe the proxy hosts only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
b14bf0cc5e |
feat(heating-can): read the One Base per-room telemetry over CAN
The Einzelraumsteuerung (8 radiator valves, 9 climate sensors, 9 underfloor actuators, 3 floor thermostats) runs on Viessmann's One Base ZigBee system. Its room and device registries and its live per-device values sit on ECU 0x6c3 (BACKENDGATEWAY) and are readable in plaintext over the E3 bus -- room temperature, the active target temperature, humidity, valve opening, demand, battery and RSSI. That replaces the ViCare cloud room sensors with a local source. 0x6c3 had never been enumerated: only 0x680, 0x6a1 and 0x68c ever had a full scan. A full read of 0x6c3 returns 155 decodable datapoints plus 35 that exist but fail the stock codec. Those 35 are the reason for the overlay. open3e declares ZigBee<N>DeviceCurrentValues as 57 bytes; a PAIRED slot answers with 68 -- the known 57 plus the parent device serial and three trailing bytes. udsoncan reads only the declared length and then treats the next two bytes, the first half of that serial, as a foreign data identifier, so every paired slot fails with "Server returned values for data identifier 0x.... that was not requested". The bogus identifier is stable per device and differs between devices because the serial does, which is why this previously read as an encrypted channel. It is not encrypted. The overlay re-declares the 29 paired slots at 68 bytes and inherits upstream's sub-type list, so the per-device-type switch -- including DeviceTemperatureSetpoint -- comes from upstream rather than being copied. Empty slots answer with 57, decode against the stock codec and are not polled. Only CurrentValues are polled. The registries are static configuration and are read on demand. Verified against payloads captured from 0x6c3 on 2026-09-17, decoded with open3e 0.7.5 (the commit pinned in the heating-can image): all 29 slots are 68 bytes, and a radiator valve yields ActualTemperature 19.3 °C, ValveOpening 0 %, DeviceTemperatureSetpoint 8.0 °C, battery 38 %, RSSI -69 dBm. The setpoint field was confirmed by changing one room in the ViCare app three times and re-reading: 8.0 -> 15.0 -> 13.0. Read-only. No write path to the per-room setpoint is known; writes to the CurrentValues DID were attempted and had no effect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Fable 5.1
|
8e9eedfd16 |
fix(netbird-proxy): watchdog restart cooldown 600 s -> 60 s
Every loss of the routing peer a proxy has pinned wedges its route
manager; the flood watchdog restarts the container within 2 s and the
public edge is back after one failed probe round. Inside the 600 s
cooldown the restart is suppressed and the URL watchdog three one-minute
rounds are the only recovery: 3.5 min of 000 on every public domain,
measured 2026-09-07 10:28 on proxy-2 ("restart suppressed, last restart
567s ago"). The morning node roll hit the same suppression on both
proxies at 06:58 and survived only because the routes recovered by luck.
Rolling reboots lose one peer every few minutes by design, so with 600 s
the second roll is always the unlucky one. A minute keeps the loop
protection (one restart per probe interval at most).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
b96260cef2 |
feat(crowdsec): arm the bouncer everywhere, drop the per-host override
proxy-2 has been enforcing since 14:21:54 without incident, so the role default moves to nftables and the host_vars override goes away rather than being duplicated for proxy-1. proxy-2's rendered config is unchanged by this, so it will not restart; only proxy-1 does. The staged rollout did its job and can be read off the deploy log: proxy-1: deployed in mode "dry-run" proxy-2: deployed in mode "nftables" and only proxy-2 restarted, which showed up as a 1,887,882-byte startup pull from its address alone. The delivery path was tested in enforcing mode with a decision on 192.0.2.1 (TEST-NET-1): both hosts had the ban 3s later as a 205-byte delta, and the revocation as a 203-byte delta 1-2s after deletion. What is still not proven is that a packet is actually dropped, and the measurement for that argues for finishing the rollout rather than waiting. Requests from banned addresses reaching Traefik stopped after proxy-2 armed — 1 at 14:02, 3 at 14:07, 22 at 14:08, 1 at 14:18, none after — but 25 of those 27 arrived in a single burst beforehand, so at this volume that is consistent with enforcement rather than evidence of it. With both hosts armed the claim becomes falsifiable instead: no request from an address in the decision set should reach Traefik at all, and a single one would prove otherwise. dry-run remains available per host through inventory/host_vars, which is what a newly added proxy should get before it enforces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
05cd7f3674 |
feat(crowdsec): arm the bouncer on proxy-2
First host goes from dry-run to enforcing. proxy-2 deliberately: it is the dual-stack member, while proxy-1 is IPv6-only and the one where every network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage, and a GitHub download that could not work at all. host_vars keeps the change to one host; proxy-1 stays on the role default. Verified before committing: ansible-inventory resolves nftables for proxy-2 and nothing for proxy-1. Adds netbird.renner.uno to capi_whitelists first, and that is not decoration. The bouncer's chain sits on the INPUT hook, the proxy hosts run no other firewall, and there is therefore no conntrack rule accepting established flows ahead of it. A ban on the management address would drop the return traffic of the proxies' own outbound control connection and take them off the mesh, with no inbound attack involved at all. The same reasoning already covers the proxy addresses, which were listed from the start. Checked at the time of writing: none of these addresses appear in the current decision set — searched all ~15k entries. That is the state today, not a guarantee; they are rented addresses on a hoster whose ranges do end up on community blocklists, which is the whole reason for listing them. The delivery path was tested end to end beforehand with a decision on 192.0.2.1 (TEST-NET-1, so no real traffic could be affected). Both bouncers received the ban within one poll cycle — a 206-byte delta against the usual 26 — and the revocation the same way, 203 bytes, ~5s after deletion. What that did not test is whether a packet is actually dropped, which dry-run cannot show. This commit is what makes that testable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
cf1237bf33 |
fix(crowdsec): unpack the bouncer on the host, not on the controller
The download moved to the controller in the previous fix and now works. The extraction that followed it does not: Failed to find handler for "/tmp/crowdsec-firewall-bouncer-v0.0.36.tgz". Command "/bin/tar" detected as tar type None. GNU tar required. The ci-ansible image ships a tar that ansible's unarchive rejects. Both host distributions have GNU tar, and only the download ever needed to move — the v6-only problem was reaching github.com, not opening a tarball. So the controller now ships the archive rather than the unpacked binary, and the host extracts it. That also keeps the artefact the checksum was verified against intact all the way to its destination, instead of verifying an archive and then copying something extracted from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
a5d8bf35a9 |
fix(crowdsec): fetch the bouncer on the controller, not on the host
The deploy failed on proxy-1: Request failed: <urlopen error [Errno 101] Network is unreachable> url: https://github.com/.../crowdsec-firewall-bouncer-linux-amd64.tgz proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com publishes an AAAA record. The host cannot reach GitHub at all, so no retry or timeout would have helped. Two alternatives do not work either. Mirroring into gitea.renner.uno is out: its public records are the internal LB VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And there is no upstream container image for this bouncer, which is expected of something that manipulates host netfilter — crowdsecurity publishes cloudflare, fastly, custom and others, but no firewall bouncer. So the controller fetches and pushes instead of the host pulling. The CI runner has v4. The checksum is still verified, now before anything reaches a host, which is the better place for it anyway. The repo had told me twice that proxy-1 is the v6-only member — in unifi_nat_hairpin.tf and in the netbird-proxy role's dhcpcd comment — and I wrote it into the PR description myself. I never asked whether a v6-only host can reach GitHub. Nothing was deployed before this: the play runs serial: 1, proxy-1 went first and failed, so proxy-2 never started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
7897b50318 |
feat(crowdsec): vault the edge bouncer key for the proxy hosts
Completes the pair: the ExternalSecret feeds BOUNCER_KEY_edge to the LAPI, this feeds the same value to the bouncers that authenticate against it. A single encrypted value rather than a whole-file vault like its siblings in group_vars/all and group_vars/edge_host. Appending to one of those means decrypting it first, which exposes every other secret in that file to add one line. encrypt_string keeps the blast radius at this value, and the plaintext never reaches a file: it goes from `bao kv get -field=key` through a pipe straight into the encryption. Scoped to netbird_proxy rather than all, because only those hosts run a bouncer. Not enrolled in the quarterly rotator, unlike crowdsec-bouncer-key. Nothing on these hosts re-reads OpenBao on a schedule, so a rotation there would leave them 401ing against the LAPI until someone ran the playbook. Rotating this one means updating OpenBao and re-running plays/install-netbird-proxy.yml, in that order. Verified with `ansible-inventory --list`: the inventory parses and the value decrypts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
22143c6a7c |
feat(crowdsec): firewall bouncer on the NetBird proxy hosts
Enforces the decisions the in-cluster CrowdSec already detects, at the point where public traffic actually enters. These hosts terminate TLS for every *.renner.uno name, so a drop here never reaches the mesh, the relay or the cluster. Detection stays where it is: the reverse-proxy produces no parseable access log and has no collection, while Traefik has a working one, and every scenario in use needs HTTP detail that does not exist at this layer. Deployed in dry-run. It pulls decisions and logs what it would block without creating a single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI decisions at once on the only external entry path, so the false-positive window has to come before the enforcement rather than after it. Arming is flipping crowdsec_bouncer_mode per host. The static release tarball, not a package. The AUR package crowdsec-firewall-bouncer-nftables depends on `crowdsec` and would pull the whole Security Engine onto hosts that need only the bouncer; the group is mixed (proxy-1 Arch, VPS-1 Debian); and the AUR would be a pattern this repo does not otherwise have. The tarball is the same artefact the AUR package downloads, pinned by version and sha256, and only the binary is unpacked — the install/upgrade/uninstall scripts beside it manage systemd and config themselves, which would leave two owners for the same files. retry_initial_connect is set, and it is not cosmetic. Without it the process exits when the LAPI is unreachable at startup, and since a clean shutdown deletes the nftables tables, the systemd restart loop would leave the host with no blocklist at all until the first successful connect. The flag exists in the code but not in the shipped example config. nftables_hooks covers input and forward. input is the traffic this exists for, because the reverse-proxy is host-networked and owns :443. forward covers the bridge-networked http-redirect sidecar, whose packets never traverse INPUT — the same Docker bridge that showed up as 172.18.0.3 in the Traefik access log. Prometheus is enabled but bound to loopback. These hosts are public and have no metrics collection path today; opening :60601 to the internet to gain a scrape target is the wrong trade. No CA file: Traefik serves Let's Encrypt for crowdsec.renner.uno, and ca_cert_path is added to the system pool rather than replacing it. Rollback is `systemctl stop` — a clean stop deletes both tables by itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
7c41a21f88 |
feat(netbird): drop the cluster-side backup of the controller VPS
Backing up an external server is that server's own business. The pull path made the cluster responsible for it and, worse, put a root key on the VPS that the cluster held — a standing inbound grant that existed only to move a SQLite file. Removed cluster-side: the netbird-backup namespace and everything in it (the nightly CronJob, the netbird-backup-ssh ExternalSecret, the 5Gi ceph-block PVC), the namespace's entry in the OpenBao ClusterSecretStore allow-list, the netbird-backup-daily Velero Schedule, and the NetbirdVpsBackupStale / NetbirdVpsBackupNeverRan alerts. Flux prunes the PVC, which destroys the 14 retained dumps (2026-08-15 through 08-28). That is intended. Velero's existing offsite copies age out on their own under the schedule's 720h TTL; no cleanup is needed there. Removed VPS-side: the dump script, and the forced-command key authorising it. Deleting the two tasks that placed them would only stop ansible managing them and leave both live on the host, so they are replaced with state: absent tasks that actually revoke them. Those two can go once the role has run against every host that ever carried them. The trmnl-daily schedule's stagger comment pointed at netbird-backup-daily for its slot; it now points at apps-daily, which is the schedule that actually precedes it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
8cde6d5127 |
fix(netbird): let the nightly dump wait out the writer
The dump ran sqlite3's .backup with no busy handler. sqlite3 defaults its busy timeout to 0 ms, so the online-backup API gave up the first time the management container held the write lock instead of waiting for it, and set -eu turned that into a failed job. Four nightly runs failed that way before this, three attempts each, every one refused within seconds of the ssh handshake. NetBird sets no journal_mode on the store — its DSN carries only _busy_timeout=30000 and cache=shared — so store.db uses the default rollback journal, where a writer's lock blocks a reader outright. Under WAL the reader would have gone through and this would never have come up. .timeout arms the busy handler on the source connection, which is the contended side. The shell's .backup never arms one on the destination and does not need to: that file is created fresh in $tmp and nothing else touches it. VACUUM INTO replaces .backup because a busy timeout on its own only turns an instant failure into a slow one. .backup copies 100 pages per sqlite3_backup_step() and SQLite restarts the copy whenever the source is written between steps, so against a steady writer it returns only once the writes stop. Measured on a rollback-journal database of the same shape, with a writer committing continuously for 20 s: .backup returned after 19 s, VACUUM INTO returned in under a second, and integrity_check on its output passed. Both failed identically with no timeout at all, which is what reproduced the production error. VACUUM INTO also compacts, so the dump lands smaller than the live file. 30 s sits well inside the puller's 600 s activeDeadlineSeconds, and the read transaction VACUUM INTO holds is covered by NetBird's own 30 s busy timeout on the writing side. The script lives on the VPS at /usr/local/bin/netbird-backup-dump and is refreshed only when this role runs, so merging this to master is what deploys it — deploy-netbird-server triggers on ansible/roles/netbird-server/**. Closes #714. Refs #712. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-bot
|
9c9e45d17d |
chore(ansible): drop the nas-1 inventory, roles and plays
nas-1 has been off for months. Everything scoped to it went with it: inventory/host_vars/nas-1.renner.uno/ vars and vault [synology_nas] in pve_hosts.yml the group's only member roles/pbs-client/ PBS backup client plays/install-pbs-client.yml role-based, hosts: synology_nas plays/install-pbs-client-synology.yml raw-module variant, same host plays/install-garage-synology.yml Garage S3, replaced by Hetzner .woodpecker/install-garage-synology.yml the pipeline that ran it The two PBS plays were already inconsistent with each other before the host went away: the role pulled aterfax/pbs-client, while the standalone play pulled proxmox/proxmox-backup-client, which has never existed on Docker Hub (404). Nothing referenced the standalone play at all. Both are moot now. pve_hosts.yml keeps its "gone from this file, and why" list, so synology_nas is recorded there alongside pve_cluster and the in-cluster migrations rather than vanishing without trace. Checked: `ansible-inventory --graph` resolves, and grep finds no remaining reference to synology_nas, nas-1, pbs-client or the garage play outside that one note. plays/archived/ is left alone — it is history, not live config. |
||
|
claude-botandClaude Opus 5
|
6ff5cb7d7f |
fix(ansible): stop rotating netbird_proxy_token — Ansible does not own both ends
Rotated turn_secret and relay_secret in production; both are correct and stay
rotated. proxy_token was in the same set and should never have been: rotating
it locked every proxy out of the management server.
The misjudgement was specific. proxy_token appears in proxy.env on the server
AND in proxy.env on each proxy, both Ansible templates, which looked exactly
like the symmetry test the other two pass. It is not: the management server
never reads it from a file Ansible writes. The token is registered in the
management DATABASE, so rotating it changed only what the proxies present.
management: proxy auth failed: rpc error: code = ResourceExhausted
desc = too many failed authentication attempts (~75/min, steady)
proxy: http: TLS handshake error ... unknown domain "home.renner.uno"
The second follows from the first — a proxy that cannot authenticate never
receives its domain configuration, so it rejects the domains it holds valid
certificates for.
Recovered by restoring the original proxy_token from the pre-rotation vault and
re-applying both roles. Auth failures went to zero and the domain errors
stopped. turn_secret and relay_secret keep their new values; both appear in
management.json.j2, so both ends really are Ansible's.
vault_netbird_proxy_token moves back into vault.yml. It was in
netbird-rotatable.yml, which this play rewrites wholesale — the next run would
have silently dropped it. Verified by re-running both deploys: changed=0.
Adds the test that would have caught this, in the play's own words: a secret is
safe to rotate here only if it appears in management.json.j2. Templating both
sides of a wire is not the same as owning both ends of a trust relationship.
Also corrects the shadow-key guard. vault.yml carried BOTH
vault_netbird_turn_secret and a bare netbird_turn_secret; the role default maps
one to the other, but a group var outranks a role default, so the bare key won
and rotating only the prefixed one would have changed nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
 k.renner
|
19462a3792 |
Merge pull request 'chore(ansible): remove monitoring-agent, which shipped to endpoints that are gone' (#629) from drop-monitoring-agent into master
Reviewed-on: k.renner/homelab#629 |
||
|
claude-botandClaude Opus 5
|
b76a948e28 |
chore(ansible): remove monitoring-agent, which shipped to endpoints that are gone
common.yml applied monitoring-agent (Alloy) to every inventory host, pointing at loki.srv.renner.uno:3100 and prometheus.srv.renner.uno:9090. Both moved into the cluster and neither name resolves any more, so nas-1, claude-ws, netbird and the proxies have been shipping logs and metrics nowhere. Repointing them was the obvious fix and the wrong one. The cluster endpoints are authenticated — loki.renner.uno returns 401 and prometheus.renner.uno sits behind an Authentik forward-auth gate — while monitoring-agent has no auth support at all. Making it work would mean per-host credentials, htpasswd entries in the cluster secret, and some answer for remote-write past the gate. Nobody reads those metrics: edge-1 is the host that matters, and edge-2 when it exists. edge hosts are unaffected. install-edge-host.yml is self-contained and deliberately never carried monitoring-agent; they ship via edge-log-shipper, which already does this correctly — authenticated push to the cluster ingress with a per-host credential. install-netbird-monitoring.yml goes too; applying monitoring-agent to netbird_server was its only content. Comments referring to the role are corrected in install-edge-host.yml, install-claude-workstation.yml, the claude-workstation role and its RUNBOOK. Three more described the old srv Loki as "firewalled off the IoT VLAN" — it is not firewalled, it is gone. The reason edge hosts use the authenticated ingress is unchanged, but the stated reason was no longer true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
9733796067 |
chore(ansible): drop the 26 vault keys nothing references
group_vars/all/vault.yml held 35 keys. Nine are referenced by the surviving
tree, all NetBird:
netbird_oidc_client_id vault_netbird_auth_client_secret
netbird_oidc_client_secret vault_netbird_datastore_key
netbird_oidc_issuer vault_netbird_proxy_token
netbird_turn_secret vault_netbird_relay_secret
vault_netbird_turn_secret
The other 26 belonged to Proxmox, n8n, owncloud, the grafana and gitea OIDC
pairs, woodpecker, SNMP, cloudflared, the UniFi Poller account and both Home
Assistant tokens — services that now run in-cluster with their own credentials,
or no longer exist at all.
Edited YAML-aware rather than line-based: at least one value was a multi-line
block, which a line delete would have corrupted.
Verified the nine survivors are byte-identical, by comparing a normalised hash
of just those keys before and after, and again after re-encrypting:
e9fbbcef4127b84289ebf32e3b99e59a68e96e275af3c8c17d8ad0414a08e790
Same hash all three times. Decryption happened in a 0600 file outside the repo,
shredded afterwards; no plaintext touched the working tree.
Removing a key here does NOT revoke the credential. Anything still live in its
target system needs revoking there — that is a separate exercise, and the Home
Assistant tokens in particular are unreferenced here but valid in HA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
c75c7accd5 |
chore(ansible): remove two more plays whose hosts are gone
Missed in the Proxmox removal. I built that list from a `hosts:` listing I had
truncated with `head -40`, so anything past the fortieth line was never
considered — a self-inflicted blind spot rather than a judgement call.
install-postgres-monitoring.yml authentik-db-1.srv.renner.uno and
prometheus.srv.renner.uno, both NXDOMAIN
configure-audiobookshelf.yml audiobookshelf.srv.renner.uno is NXDOMAIN;
audiobookshelf runs in-cluster
The postgres-exporter role goes with them — install-postgres-monitoring.yml was
its only consumer.
Found by cross-referencing every vault key against the surviving tree rather
than by looking at plays again: vault_postgres_exporter_password came back as
"still referenced", which was only true because a dead play still referenced it.
That check now leaves the group_vars/all vault holding nine keys anything uses,
all NetBird, and twenty-six nothing does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
k.renner
|
98e4cb77bd |
Merge pull request 'chore(ansible): remove Proxmox and the guests that lived on it' (#625) from ansible-drop-pve into master
Reviewed-on: k.renner/homelab#625 |
||
|
claude-botandClaude Opus 5
|
48130ea4f1 |
fix(ansible): drop the community.general dependency and harden the guard
The play generated secrets with community.general.random_string. The CI image (ci/images/ansible/Containerfile) installs only community.proxmox and community.docker, so the play failed at its first task when run through the project's own tooling. It resolved on a workstation that happened to have the collection, which is exactly why verifying it there proved nothing. Replaced with ansible.builtin.password, which needs no collection and produces the same thing: 48 alphanumerics, generated without persisting (path /dev/null). Also hardens the shadow-key guard. It piped `ansible-vault view` into `grep -c` with failed_when: false, so a failed decrypt — wrong password, moved file — sent empty input to grep, which reported zero matches. Indistinguishable from a clean vault, meaning the guard would have waved the rotation through precisely when it knew least. A preceding task now proves the vault is readable, and the assertion requires both that proof and the zero count. Corrects the openbao rotation path in the header comment; it is under kubernetes/. Caught by the review-bot on #626. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
52899ecb64 |
fix(ci): remove update-all, which invoked a deleted playbook
.woodpecker/update-all.yml ran `ansible-playbook -i inventory/ plays/update.yml`
on a cron. This branch deletes that playbook, because every playbook it
imported targeted Proxmox hosts or guests that no longer exist, so the next
scheduled run would have failed outright.
Removing the pipeline rather than repointing it: there is nothing left for it
to update. Its imports were update-vm, update-lxc, update-authentik,
install-n8n, install-grist, install-music-assistant, update-docker and
update-pve-rolling — all removed here, all targeting hosts that no longer
resolve.
Swept every pipeline for references to deleted plays. The five still named
(install-claude-workstation, install-garage-synology, install-netbird-client,
install-netbird-proxy, install-netbird-server) all survive.
Also from the review:
common.yml `all:!pbs` was a no-op I introduced when simplifying the
old exclusion list — no pbs group exists. Now plain
`all`.
qemu-guest-agent dropped the `pve_cluster`/`proxmox_nodes`/
`proxmox_all_lxc` group guards; those groups are gone,
so the conditions were always true.
Caught by the review-bot on #625.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
3c6278b144 |
fix(ansible): guard against vault.yml shadowing the rotated keys
group_vars/<group>/ files merge alphabetically and the last one wins. netbird-rotatable.yml sorts before vault.yml, which still defines vault_netbird_turn_secret, vault_netbird_relay_secret and vault_netbird_proxy_token — verified, all three are present. The rotation itself would have worked, since the plays pass the new values explicitly and play vars outrank group vars. The damage came later: the next ordinary run of install-netbird-server.yml or install-netbird-proxy.yml would read the stale value from vault.yml and push it back out, breaking server<->proxy auth. Those deploys run unattended on changes to the roles, so it would have surfaced on something unrelated to secrets. Adds a pre-flight check that counts — never prints — matching keys in vault.yml and refuses to rotate while any remain, plus the one-time migration step in the header. Caught by the review-bot on #626. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
b3f785645f |
feat(ansible): play to rotate the NetBird secrets Ansible owns
Rotates the three shared secrets where Ansible controls both ends, so the two
sides cannot disagree:
vault_netbird_turn_secret turnserver.conf (coturn) + management.json, one
role, one host
vault_netbird_relay_secret management.json on the server + relay.env on the
proxies
vault_netbird_proxy_token proxy.env on both
Opt-in, because rotation is not idempotent and briefly interrupts the mesh:
ansible-playbook plays/rotate-netbird-secrets.yml -e rotate_confirm=yes
New values are written to a dedicated encrypted file rather than edited into
the main vault. The plaintext is rendered at 0600 with a .new suffix, encrypted,
and only then moved into place, so a failure at any step cannot leave readable
secrets where the inventory would load them. The play asserts the installed
file carries the $ANSIBLE_VAULT header before proceeding.
Two are deliberately excluded and the play says why:
vault_netbird_datastore_key encrypts the datastore at rest; changing it
without dump/restore makes existing data
unreadable
vault_netbird_auth_client_secret its other half is in Authentik, so rotating
only the consumer breaks login
These are Ansible's own credentials, not copies of the cluster's. OpenBao holds
separate credentials for some of the same systems, deliberately different values
— one consumer, one credential, independently revocable. This play does not
touch OpenBao and is not meant to.
Proxies roll one at a time so a relay stays reachable. The server is
reconfigured first, so there is a short window where a not-yet-updated proxy
fails to authenticate; clients reconnect and peer-to-peer tunnels are not torn
down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
712a816f88 |
chore(ansible): remove Proxmox and the guests that lived on it
Nothing here had a target left. Every removal is backed by a probe rather than
an assumption:
pve_cluster (jupiter/neptune/saturn) run Talos; no SSH at all
authentik-db-1/2, authentik-app-1/2 NXDOMAIN
grafana.srv, vault.srv, paperless.srv gone (grafana/paperless resolve but
answer neither ping nor SSH)
n8n.srv, grist.srv, wiki.srv NXDOMAIN
owncloud.srv, ical-proxy.srv, loki.srv NXDOMAIN
proxmox_* dynamic groups api.proxmox.yaml queried
jupiter.renner.uno:8006
24 plays, 14 roles and the Proxmox dynamic inventory go. update.yml goes with
them: every playbook it imported is in that list, so the aggregate updater had
no fleet left to update.
Kept, because their hosts answer: netbird server and proxies, nas-1, edge-1 and
its capability groups, claude-ws. common.yml and install-node-exporter.yml lose
exclusion clauses naming groups that no longer exist.
NTP IS NO LONGER HIGHLY AVAILABLE, and this change records that rather than
causing it. ntp_cluster was jupiter (MASTER, prio 100), saturn (90), neptune
(80) and edge-1 (BACKUP, 70); the conversion took the first three. A single
VRRP member holds the address regardless of configured state, so fd40::123 is
still served — verified answering — but by one thin client with nothing behind
it. install-ntp-ha.yml still carries the four-host priority map, now describing
hosts that do not exist. Restoring redundancy needs a second chrony host that
is not a Talos node.
The gatus role is removed entirely, which supersedes the narrower edit in the
DNS cleanup that dropped its Proxmox DNS check.
Left alone: plays/archived/ still references pve_cluster, which is what an
archive is for. paperless-migration stays because an archived play uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
56bb47e483 |
chore: remove the decommissioned Proxmox DNS server and its migration script
The Proxmox DNS HA VIP (10.2.40.53) was decommissioned after it dead-ended
non-Ingress resolution and broke ~1374 Home Assistant entities on a pod restart
(2026-07-14). With Proxmox itself now gone, the implementation and everything
pointing at it are dead weight — and one piece was actively dangerous.
Removed:
dns-server/ the Go implementation, its systemd unit,
and a 7.4 MB checked-in binary
opentofu/scripts/remove_unifi_clients.sh
the "Proxmox DNS (VIP)" Gatus check in the ansible role
remove_unifi_clients.sh is the one worth calling out. It instructs the reader
to deploy module.proxmox_dns and point UniFi DNS at 10.2.40.53 — precisely the
configuration that caused the outage. It also works by `tofu state rm`, which
orphans resources rather than removing them. It was already marked deprecated
in its own header; anyone who ignored that and followed it would have
reintroduced a known failure.
The Gatus check queried k8s-1.srv.renner.uno against 10.2.40.53 every 30s. It
belongs to the ansible gatus role, which targets a Proxmox LXC that no longer
exists, so nothing was running it — but the check itself asserts the decommis-
sioned server should answer, which is the opposite of the intended end state.
Deliberately kept: the three comments in bootstrap.sh, coredns-config and
k8s-gateway that record 10.2.40.53 as DECOMMISSIONED and say not to reintroduce
it. They carry the outage date and blast radius, and are the reason this keeps
not happening again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
8e5b91e6d6 |
chore(ansible): retire the monitoring plays for decommissioned hosts
Prometheus and Loki run in Kubernetes -- statefulsets prometheus-kube-prometheus-stack-prometheus and the Loki release, both up 93 days. The standalone VMs these plays targeted are gone: neither prometheus.srv.renner.uno nor loki.srv.renner.uno resolves at all, and the proxmox_logging group install-log-aggregation.yml targets is not in the inventory. So these rendered scrape configs, exporter installs and log shippers for hosts that do not exist. Removed: plays/install-prometheus.yml -> prometheus.srv.renner.uno (gone) plays/install-blackbox-exporter.yml -> prometheus.srv.renner.uno (gone) plays/install-loki.yml -> loki.srv.renner.uno (gone) plays/install-log-aggregation.yml -> proxmox_logging (no such group) roles/prometheus-server, snmp-exporter, pve-exporter, unifi-poller, roles/blackbox-exporter, loki-server, promtail Each role was consumed only by the plays above; promtail was referenced by nothing at all. Verified no dangling references remain. plays/update.yml imported install-log-aggregation.yml, so deleting it alone would have broken the fleet update play. That import is removed and update.yml re-syntax-checked. This also retires #473, which changed IPv4 literals to FQDNs in prometheus.yml.j2 and nodes.yml.j2. That was a change to a template nothing renders. Its PR body described the instance-label consequences for dashboards and alert rules, which was fiction -- no dashboard reads those labels. When it would not apply I reported it as blocked on Proxmox API credentials; the actual reason was that the target host had not existed for months, and `--list-hosts` returning 0 was the evidence I misread. Worth checking separately, not assumed here: whether anything scrapes the PVE node_exporter and chrony_exporter today. install-node-exporter-pve.yml still installs them on pve_cluster, but the config that pointed a Prometheus at them was in the templates just deleted. If that was the only reference, those metrics have been dark since the VM went away -- and the FQDN change in #473 would have masked the question rather than raising it. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
1b7e3dc100 |
fix(heating-can): stop the native services before removing their units
The migration did not stop anything. It ran stop + disable + mask in one
call with failed_when: false, then deleted the unit files. Masking a unit
whose file still exists conflicts, so the call failed, failed_when hid the
failure, nothing was stopped -- and the next task deleted the unit files out
from under two running processes.
That left systemctl is-enabled reporting "not-found", which looks clean,
while is-active still said "active". I reported the cutover successful on
the strength of the former.
On 2026-08-10 the result was two open3e instances issuing UDS requests on
the same CAN bus from 19:28 to 19:37, taking over each other's
open3e-edge MQTT session -- the exact failure the block's own comment said
it existed to prevent.
Fixed in three parts:
- Stop and disable FIRST, and do not swallow the result. Only units that
actually exist are touched (stat first), so a fresh host does not need
an error hidden to get through.
- No masking. It writes a symlink at the same path the next task removes,
so the two cancel out. With the unit files gone there is nothing to
start.
- Assert the invariant that actually matters -- no native process on the
CAN bus -- rather than inferring it from systemd reporting success.
pgrep, then fail loudly with the surviving command lines and why it
matters.
Verified against the live host: ok=15 changed=1 failed=0, the assert passing
with "No native CAN processes remain; safe to start the container", and the
run now idempotent (the single change is the compose up, not the teardown).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
150a83ee38 |
fix(edge): make the capability-group invariant structural, and unstale a note
Both from review on #475. The rule "a host in a capability group must also be in [edge_host]" was enforced by a comment, in a file that already uses :children a few lines below for ntp_cluster. The three capability groups are now children of edge_host, so the relation is structural: a host added only to [heating_can] still resolves group_vars/edge_host, where the local MQTT contract lives, and cannot silently come up without edge_mqtt_host and friends. That matters more than it looks. The MQTT contract only moved to group_vars in this PR because splitting the plays exposed that the roles could not render without each other's defaults -- so an edge host outside [edge_host] would hit exactly the failure that change was made to fix. Verified all four groups still resolve to edge-1, and the redundant ansible_user on the capability entries is dropped: it is inherited from the [edge_host] declaration rather than repeated three times. Also unstaled a Day-2 note that still read "the pip task tracks the git default branch (pin edge_open3e_pkg to a tag if that ever bites)". Both services are pinned to commits now, deliberately, and pinning to a tag is not available for open3e -- it publishes none. Replaced with the actual workflow: the weekly update-audit reports drift, bump the pin and re-run. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
0b845e7c76 |
refactor(edge): split the plays by capability, not just the roles
Splitting edge-host into single-concern roles left the coupling one layer
up: install-edge-host.yml was still one play applying all eleven roles to
one group. A host in [edge_host] got the Z-Wave gateway and the CAN services
whether or not it had the hardware -- the monolith again, in the play.
Inventory groups now state which hardware a host physically has, and each
playbook targets the group for its own concern:
[edge_host] install-edge-host.yml base: flash-wear, log
shipper, MQTT bridge, VLAN,
docker
[heating_can] install-heating-can.yml Innomaker USB2CAN
[zwave_gateway] install-zwave-gateway.yml Nabu Casa ZWA-2
[power_meter] install-power-meter.yml FTDI meter head
No flags and no `when:` on the roles. Moving a stick to another box is a
one-line inventory change, which is what it should be. A host with nothing
attached is simply absent from the capability groups -- which is what lets a
second thin client join as an NTP peer without being handed peripherals it
does not have.
The split exposed a circular dependency the single play had been hiding: the
local MQTT contract was scattered across two roles' defaults, in both
directions. mqtt-bridge's config needs edge_open3e_mqtt_user, because the
bridge authenticates to the cluster broker AS that user; heating-can and
power-meter-sml need edge_mqtt_host, because that is where they publish.
Neither role could render without the other's defaults, and it only ever
worked because one play loaded every role at once.
Those four values are the interface between the broker and its clients, not
the property of either side, so they move to group_vars/edge_host -- where
the matching secrets (edge_open3e_mqtt_password, edge_sml2mqtt_mqtt_password)
already lived. Removed from the role defaults so there is one source of
truth. Each role keeps what is genuinely its own: mqtt-bridge the cluster
endpoint and CA path, heating-can the datapoints and poll interval.
Also dropped: the legacy shared-compose teardown pre_tasks. That migration
completed on the 2026-08-08 apply (verified: /opt/edge-host/docker-compose.yml
is gone), and carrying migration cruft in a base play would be the same
mixing this commit is undoing.
Verified by --check --diff against the live host, all four plays:
install-edge-host ok=33 changed=0 failed=0
install-zwave-gateway ok=8 changed=0 failed=0
install-power-meter ok=4 changed=0 failed=0
install-heating-can ok=18 changed=4 failed=0
Three of four are fully idempotent. heating-can's four are the known open3e
pip churn plus the E3onCAN venv, which reports changed only under check mode
(it was ok in the real apply) -- and their two restart handlers.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
f9848d2bf5 |
fix(heating-can): correct the overlay path, secret mode and writable CWD
Review on #476 found a real regression plus three things worth fixing before this reaches edge-1. Blocking: edge_open3e_devices.vx3.dpList still pointed at the old host path /opt/edge-host/open3e/vx3plus.py. The compose file mounts that overlay at /config/vx3plus.py and nothing mounts anything under /opt/edge-host inside the container, so open3e would have looked for it at a path that does not exist -- silently falling back to the stock vx3 datapoint list and dropping the battery-module DIDs (vx3.1577-1582 and friends) the overlay exists to add. Now /config/vx3plus.py. Worth noting the build-time self-test could not have caught this: it proves the binaries execute, not that devices.json resolves against the mounted files at runtime. Writable working directory. WORKDIR was /opt, which the COPY layers create root-owned, while the process runs as open3e. udsoncan writes its log file into the CWD -- the native install gave open3e's directory `owner: open3e` for exactly that reason, and I dropped that when containerising. Now a dedicated /var/lib/open3e, owned by the service user. Secret mode. MQTT_PASSWORD was inline in docker-compose.yml, which is rendered 0644. The env file this replaced was 0600, and the sibling roles on this host keep secrets in a 0600 file beside a world-readable compose file. Restored: credentials move to heating-can.env at 0600, referenced via env_file. edge_e3oncan_mqtt_format had become dead -- both services were fed MQTT_FORMAT from edge_open3e_mqtt_format. The two default to the same string so behaviour was unchanged, but overriding it did nothing. Wired through as E3ONCAN_MQTT_FORMAT, which supervisord passes to e3oncan only. Also from review: dropped the registered-but-unread heating_can_native_stopped, and the migration now removes the emptied {{ edge_base_dir }}/open3e and /e3oncan directories rather than leaving them behind. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
96fcb426aa |
build(heating-can): one container, both readers supervised inside it
Reverts my own reinterpretation. The instruction was to put both in one container because they belong together; I argued against it on crash-isolation grounds, then read the correction "two services, one image" as agreement with my version rather than as a restatement of the original. It was not. A container runs one PID 1. With two processes that PID 1 is a supervisor, which is how a container handles multiple services. supervisord now runs both readers: it restarts either on its own if it dies, reaps children and forwards shutdown -- none of which a bare process as PID 1 can do for a sibling, which was the actual answer to the concern I raised. Also gone: the dispatch entrypoint and the argv in the compose file. Both were the same mistake in different places -- the invocation living outside the artifact. open3e's and E3onCAN's command lines are an implementation detail of this image now; supervisord builds them from environment variables it expands itself. Nothing outside the image knows either CLI, so an upstream flag rename cannot break the compose file or an Ansible template. The compose file supplies configuration and nothing else: one service, one container, environment only. The credential env file is dropped -- the credentials are environment already. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
56a9859e50 |
refactor(heating-can): run the image, install nothing on the node
Nothing is installed on the host any more. The image is built in CI with both sources pinned, published to the registry, and edge-1 pulls and runs it. That is the whole point of building it -- installation happens at build time, once, not on every converge. Removed from the node: both venvs, the E3onCAN checkout, and the two systemd units. Kept and still rendered: devices.json, the vx3 datapoint overlay and the broker credentials, now bind-mounted into the containers rather than read from a venv working directory. Two services from one image, as before: open3e drives the Vitocal/VX3 over UDS, e3oncan listens passively for the E380 meter. Separate containers so either can crash and restart without a supervisor as PID 1. The migration stops AND masks the old units before the containers start. Masked rather than disabled because two open3e instances issuing UDS requests on one CAN bus is the failure mode to avoid, and a stray `systemctl start` or leftover dependency should not be able to bring the native one back alongside the container. No missing-device tolerance here, unlike the zwave and meter roles: can0 is this host's purpose. If it is absent the heating telemetry is down and the run should say so rather than pass quietly. heating_can_image replaces edge_open3e_pkg / edge_e3oncan_repo / edge_e3oncan_version. The source pins move into the Containerfile, so what is deployed here is a tag -- and `docker compose up -d` on an unchanged tag is a genuine no-op, which is what removes the restart-per-converge that pinning the pip URL could not. Verification is limited and worth stating: plays/install-heating-can.yml lives in #475 and is not on this branch, so only the master-side play syntax-checks here. A --check run cannot exercise this at all -- command tasks are skipped under check mode, so neither the pull nor the compose up would run. It cannot be meaningfully verified until the image has been built and #475 is merged. Depends on #475 (the capability-split playbooks, and the MQTT contract moving to group_vars). This branch is cut from master, so heating-can's defaults will conflict there; #475 should merge first and this be rebased onto it. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
6cfa63f6f3 |
fix(edge): keep the docker log-driver config out of the shared role
Review on #474 caught that the previous commit put the daemon.json task in docker-install, which is shared by ten plays: install-netbird-server, install-netbird-proxy, install-n8n, install-authentik-ha, install-owncloud, install-gatus, install-ical-proxy, install-grist and install-music-assistant. None of those hosts has an /etc/docker/daemon.json today -- the template and the default were both new here -- so the next routine run of any of those plays would have created one, reported changed, fired the Restart docker handler, and bounced every container on that host. Unannounced restarts across authentik, n8n, owncloud, grist, netbird, gatus, music-assistant and ical-proxy, from a change about edge-host bootstrapping. The --check verification could not have caught it: it only ever exercised install-edge-host.yml, and the damage was in the other nine plays. The comment I wrote there was the tell. It said "the services that use this host run with network_mode: host" -- true of edge-1, false of most consumers of that role. A justification that only held for the one host I was looking at, written into a shared file. Moved to edge-flash-wear, which is edge-host-only and already owns this rationale: routing container stdout to the journal is flash-wear handling, not generic docker setup. The networking aside is dropped -- in a flash-wear role the only thing worth saying about daemon.json is why logging goes to the journal. docker-install is now byte-identical to master (git diff origin/master -- roles/docker-install/ is empty). Two details the move needed: /etc/docker is created explicitly, because edge-flash-wear runs before docker-install on a first bootstrap and the directory will not exist yet; and the Restart docker handler fires at end-of-play, by which time docker is installed. Verified: ok=67 changed=25. The changed count is unchanged from before the move, and no docker task appears in it -- daemon.json renders byte-identical to what the old role produced, so it reports ok and the handler never fires on edge-1. Also from review: noted on the play that heating-can, zwave-gateway and power-meter-sml each declare edge_base_dir in their own defaults so they stay usable standalone, that play vars win, and that the two must be kept in step. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
9cc20251fd |
fix(edge): retire the legacy compose project before any role starts a container
Review on #474 found two blocking defects, both in exactly the place the --check verification is blind: command and shell tasks are skipped, not simulated, under check mode. So "verified with --check --diff" covered everything except the two things that would actually have broken. 1. Ordering: a real run would have aborted. The legacy-project teardown lived in power-meter-sml, which the play runs last. zwave-gateway runs before it and does `docker compose up -d` for a container named zwave-js -- while the old shared project still owns that name. Docker enforces container names globally, not per project, so the command fails with "Conflict: container name already in use". That is not the missing-USB-device string the task tolerates, so the task fails and the play aborts before ever reaching the cleanup. Moved to pre_tasks, which is its correct home anyway: it is a migration step for both services, and putting it inside either one meant whichever ran first hit the conflict. The play now also declares edge_base_dir and edge_legacy_compose in its own vars rather than borrowing them from a role's defaults, since pre_tasks must not depend on role default resolution. And it removes the now-orphaned zwavejs.env left at the old shared project root. 2. Dangling handler. The libeatmydata task still notified "Restart edge compose services", a handler that was deleted with the old role. Masked today only because `creates:` makes the task report ok -- but the role's own comment says to delete that .so when the Alpine base bumps, and the next time it does, Ansible would fatally error with "handler not found". Now notifies "Restart zwave-js". Swept every notify in the new roles against every defined handler; that was the only unresolved one. Also from review: kubernetes/bootstrap.sh referenced the deleted role. Repointed to ansible/docs/edge-host.md. Re-verified: ok=66 changed=25. The teardown moved from the power-meter-sml group into pre_tasks and gained the orphaned-env removal, so the count goes 5 + 5 + 14 -> 5 pre-existing + 6 migration + 14 header repointing. Nothing else moved. The limit is worth stating plainly: this re-run still cannot prove the teardown works, for the same reason it missed the bug. The first real apply is the test. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
2d9c3ba213 |
refactor(edge): split the edge-host role into one role per concern
edge-host was a single role doing eight unrelated jobs: flash-wear
mitigation, log shipping, VLAN networking, an MQTT bridge, a CAN interface,
the heating CAN services, a Z-Wave gateway and a power-meter reader.
That is not just untidy. It is what blocked a second thin client from
joining as an NTP peer: the host-generic work (journald to RAM, Alloy, the
VLAN sub-interface) could not be applied without also handing the box a
Z-Wave radio and a CAN bus it does not have. The role bound its services to
specific USB device paths, so a second host would have tried to start
containers for hardware that is not there.
Now:
edge-flash-wear journald -> RAM, noatime/lazytime, TRIM, eMMC health
edge-log-shipper Alloy: journal -> Loki
edge-vlan tagged VLAN sub-interface
mqtt-bridge loopback mosquitto -> cluster broker over TLS
socketcan can0 only
heating-can open3e (Vitocal/VX3) + e3oncan (E380 meter)
zwave-gateway zwave-js-ui, own compose project
power-meter-sml sml2mqtt, own compose project
The E380 reader lives with open3e rather than in its own role: it is part
of the same heating installation -- it feeds the heat pump's own energy
management -- not a separate system that happens to share a bus. can0 does
stay separate, because an interface is not the same concern as the services
consuming it.
Each role installs its own packages, so it stands alone. An edge host with
no peripherals takes the first five and none of the rest.
docker-daemon.json moves to docker-install with its own docker_log_driver
variable and Restart docker handler, since configuring the docker daemon was
never an edge-host concern. Handlers split ten ways; defaults split per
role; all sixteen templates and files moved with git mv.
Two deliberate behaviour changes on the node, both accepted:
1. Compose split. zwave-js and sml2mqtt shared a compose file, so touching
either recreated both. They are now separate projects in their own
directories. container_name is fixed, so the old project must release the
names first -- a guarded one-time task downs it and removes the file.
First apply costs a brief down/up on both. Data paths are unchanged; the
zwave store is not migrated.
2. Header repointing. Templates and scripts carried "Managed by
roles/edge-host" headers, which render onto the host -- they would have
sent an operator to a role that no longer exists. Updating them changes
file content, so it restarts the services that consume those files.
Verified by --check --diff against the live host. 24 changed items, and
every one is accounted for:
5 pre-existing, unchanged from the baseline before this work:
common timesyncd; open3e venv; E3onCAN venv; and the open3e /
e3oncan restart handlers. The venv tasks install from a git URL,
which pip cannot verify, so they report changed on every run --
a separate defect, not introduced here.
5 the compose split: zwave env + compose and sml compose to their
new paths, plus the two legacy-project retirement tasks.
14 the header repointing: journald conf, eMMC health script, alloy
config, mosquitto bridge config, 80-can0.network, open3e overlay /
env / unit, e3oncan unit, zwave backup script + env, sml2mqtt
config -- and the resulting restarts of journald, alloy, mosquitto,
open3e and e3oncan.
Nothing changed outside those three groups.
One consequence check mode cannot show: 80-can0.network.j2 changed, so the
"Reload networkd" handler fires on a real run and reconfigures can0,
briefly dropping the bus under open3e and e3oncan. It is absent from the
check-mode output only because shell handlers are skipped there. The VLAN
templates did NOT change, so the riskier reconfigure of the parent uplink
on this headless host is not triggered.
Also fixed in passing: the keepalived-ntp guard told operators to recover
with `--limit <host> --tags vlan40`, and referred to the edge-host role.
Both corrected. The role README becomes ansible/docs/edge-host.md -- it
documents the host, not any single role now.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
07faf0ae69 |
fix(edge-host-iso): carry the bootstrap fixes into the ISO side too
Review on #474 caught that the previous commit fixed the procedure in ansible/roles/edge-host/README.md and left the same two defects in the two places an operator actually reads them first. That happened because my grep for the stale patterns ran with the working directory set to ansible/, so `images/edge-host-iso/README.md` was a path that did not exist. grep returned nothing and I read that as clean. A false negative from a bad path, not a clean sweep. Fixed now, verified from the repository root: airootfs/usr/local/bin/edge-install Printed the superseded two-command sequence at the console immediately after imaging -- the very first instructions anyone sees, telling them to run common.yml with -l edge_host. Now prints the single unrestricted run, plus the two steps that precede it (confirm the host resolves by name, add it to [edge_host]). README.md Described the image as being for `edge-1.srv.renner.uno`, a name that does not resolve, and repeated the same two-command bootstrap. Both replaced. This file is the link target of step 2 in the ansible-side README, so following the procedure end to end landed straight back on the defect #474 claims to close. Also corrected while there: the ISO README called the target "the bare-metal N100". edge-1 is a Wyse 3040 (Atom x5-Z8350), and the installer text further down already says it "covers the Wyse 3040". Rewritten to describe the image as not tied to one box or model, since the installer autodetects the target disk -- which is what makes it usable for the second thin client. Also from review: a comment on why `become: true` is set when the connection is already root. Not taken -- ignore_unreachable, which common.yml sets. The review's own reasoning applies: Ansible already skips an unreachable host and continues with the others, so it adds nothing here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
e288ebe39d |
fix(edge-host): make the bootstrap self-contained and the runbook followable
The documented base-install procedure had three defects, two of which are
standing house rules and one of which simply does not work.
1. It required a DHCP reservation.
"Add the DHCP reservation/DNS entry for <host> on the UDM" was step 3 --
a reservation as a documented precondition. Reservations are off the
table here. The UDM registers DHCP leases in DNS on its own, so the
record appears without one, and the host is identified by name
everywhere: inventory, plays, README. Its address is never recorded.
2. It required --limit, because the plays were split wrong.
Step 4 was:
ansible-playbook plays/common.yml -l edge_host
ansible-playbook plays/install-edge-host.yml
The --limit was load-bearing, not sloppiness: common.yml is
`hosts: all:!...`, so running it unrestricted while bootstrapping a
single new box would reconfigure the entire fleet. The split left no
other option. But --limit narrows the target list rather than staging
anything, and leaves every other host outside convergence with no report
that they were skipped.
Fixed structurally rather than by deleting the flag: install-edge-host.yml
now carries `certificates` and `common` itself -- the parts of common.yml
that apply to an edge host -- so the bootstrap is one unrestricted run
over the edge_host group. No second command, no limit.
Not carried over: qemu-guest-agent (these are bare metal, not VMs) and
monitoring-agent (Debian-oriented; edge hosts ship logs via edge-alloy).
common.yml's loki/prometheus vars go with monitoring-agent and are not
needed -- verified that the common role does not reference them.
3. It named a host that does not exist.
The README said `edge-1.srv.renner.uno` in both the base-install steps
and the Z-Wave cutover checklist. The host is `edge-1.iot.renner.uno`;
the .srv name does not resolve, so anyone following the runbook hits a
dead name at step 3.
Verified idempotent against the live host before committing:
`ansible-playbook plays/install-edge-host.yml --check --diff` over the whole
edge_host group -> ok=53 changed=5 failed=0, with all five changes inside
the edge-host role (open3e venv, E3onCAN checkout/venv), which was already
in this play. certificates and common contributed no changes at all.
Also checked the common role branches only on Debian/RedHat os_family, so
its OS-specific tasks skip cleanly on Arch.
Prompted by preparing a second Wyse 3040 as an NTP peer -- this is the
procedure that would have been followed to build it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5
|
17c12fe13f |
refactor(prometheus): address the PVE nodes by name in scrape targets
Follow-up to #472, which changed the inventory but left the same three IPv4 literals in the monitoring config and a runbook. Raised in review there. prometheus.yml.j2: - pve job targets -> FQDNs - chrony job targets -> FQDNs, and restructured to one static_config per host so each carries its own hostname label The chrony restructure is the substantive part. The previous form listed all three targets together and recovered the hostname in relabel_configs by regex-matching __address__ against each node's IP: - source_labels: [__address__] regex: '10\.2\.40\.201:.*' target_label: hostname replacement: 'jupiter' That silently coupled the labels to the addressing. Swapping a target for a name would have left every regex unmatched and the hostname label simply absent, with the scrape still succeeding and nothing failing -- a find-and-replace on the addresses alone would have broken the labels quietly. Per-host static_configs remove the coupling entirely. nodes.yml.j2: targets -> FQDNs. Already one entry per host with an explicit hostname label, so nothing else to change. claude-workstation/RUNBOOK.md: the kubeconfig command pointed --server at https://10.2.40.201:6443. That is jupiter, a Proxmox host, not a control plane node -- stale from when k8s ran as VMs on it. The cluster's endpoint is https://k8s.renner.uno:6443 (kubernetes/talos/patches/cluster.yaml), and the name is also what the API server certificate carries as a SAN, so an address there fails TLS verification even when reachable. The documented command was very likely broken as written. Deliberately NOT changed: kubernetes/apps/pve/endpointslice.yaml. An EndpointSlice takes literal addresses; hostnames are not valid in that field, so those three are correct as they are. Note this changes the `instance` label on the pve job from an address to a name, since the relabel chain feeds __address__ into __param_target and on to instance. Anything selecting on the old value needs updating. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
k.renner
|
332950b564 |
Merge pull request 'refactor(inventory): address the PVE nodes by name, not by IPv4 literal' (#472) from inventory-pve-hostnames into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#472 |