feat(ntp): extend NTP roles to edge-1, restore keepalived-ntp #315
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#315
Reference in New Issue
Block a user
Why
The three PVE hosts currently serve NTP to ~150 LAN clients: chrony with
allow 10.2.0.0/16on each, keepalived holding VIP10.2.40.4, handed to clients by UniFi DHCP option 42 on VLAN 40. Talos can run neither daemon, so the planned bare-metal migration deletes this role outright.There is a bootstrap circularity to avoid here.
talosctl get timeserverspecon k8s-2 returnstimeServers: [10.2.40.4], layer: operator— the Talos nodes learn NTP from DHCP, and there is nomachine.timeblock anywhere in the repo. So the cluster's time source is a VIP that only exists because of the hypervisors being retired, and Talos gates boot on time sync. The time source has to be on the LAN (not public internet, which fails during a WAN outage) and outside the cluster (not an in-cluster pod, which cannot come up before etcd). edge-1 satisfies both.What
chrony-ntp-servergains Debian/Arch support. Three things differ and each would have broken on edge-1:/etc/chrony/chrony.conf/etc/chrony.confchronychronyd_chronychronyArch also ships no
/var/lib/chronyor/var/log/chrony, deferring them totmpfiles.d; the role now creates them explicitly rather than depending on pacman hook ordering.systemd-timesyncdis stopped where present — edge-1 runs it today and it contends with chrony for the clock.keepalived-ntpis restored.install-ntp-ha.ymlreferenced it but the role was absent frommaster, so the playbook could not resolve at all. Recovered from thehomelab-migration-summaryworktree with three fixes:systemctl is-active chrony. On Arch that unit does not exist, so every check fails, parking the host in permanent FAULT so it never takes the VIP. This would have surfaced only at cutover, with the PVE nodes already gone. Unit name is now templated.os_family. Unlike chrony, config path and unit name are identical across distros here, so nothing else changed.net.ipv4.ip_forward=1— correct on the hypervisors, but on edge-1 it would route between IoT VLAN 30 and service VLAN 40 once the tagged interface exists, bypassing the UDM's inter-VLAN policy.edge_hostkeeps onlyip_nonlocal_bind; VRRP does not need forwarding.Scope of the two roles
Deliberately different, and this is the main thing to review:
10.2.30.108);allow 10.2.0.0/16plus inter-VLAN routing covers every client. No network reconfiguration, so this lands as a working second NTP server today.ntp_keepalived_enabled, currentlyfalsefor edge-1. VRRP needs layer-2 adjacency on VLAN 40 and the host has noenp1s0.40— the switch port now permits the VLAN, but/etc/systemd/network/has no matching.netdev/.networkpair.edge-1's keepalived priority is 70, the lowest, so it stays BACKUP while the PVE nodes exist and wins the VIP by default once they are gone — no config change at cutover.
Verified
ntp_clusterresolves to jupiter, neptune, saturn, edge-1./vardirs confirmed against the package file list, not assumed.Not verified — please read before merging
ansible-inventoryneeds the vault password foredge_host/vault.yml.enp1s0.40is a separate, riskier step.20-wired.networkmatchesType=etherbroadly and the box is headless; worth doing with console access.keepalived_ntp_auth_passis plaintext and unvaulted (carried over as-is). VRRP PASS auth crosses the segment in the clear regardless, so vaulting is cosmetic — flagging rather than silently carrying it.homelab-migration-summaryworktree's CLAUDE.md mandates CI-only execution, but there is no NTP pipeline in.woodpecker/and themastercheckout has no CLAUDE.md at all. If the CI-only rule stands, a pipeline is needed.The three PVE hosts currently serve NTP to ~150 LAN clients via chrony with keepalived holding VIP 10.2.40.4, handed out by UniFi DHCP option 42. Talos cannot run either daemon, so the planned bare-metal migration removes this role entirely. edge-1 is brought into the NTP group to take it over. chrony-ntp-server gains Debian/Arch support. The two distros differ in three ways that would each have broken on edge-1: config path (/etc/chrony/chrony.conf vs /etc/chrony.conf), unit name (chrony vs chronyd), and service user (_chrony vs chrony). Arch also ships no /var/lib/chrony or /var/log/chrony, deferring them to tmpfiles.d, so the role now creates them explicitly rather than relying on pacman hook ordering. systemd-timesyncd is stopped where present, since edge-1 runs it today and it contends with chrony for the clock. keepalived-ntp is restored -- install-ntp-ha.yml referenced it but the role was absent from master, so the playbook could not resolve. Recovered from the homelab-migration-summary worktree with three fixes: - The health check hardcoded `systemctl is-active chrony`. On Arch that unit does not exist, so every check would fail, parking the host in permanent FAULT so it never takes the VIP -- surfacing only at cutover, once the PVE nodes were already gone. The unit name is now templated. - Install branches on os_family. Config path and unit name are identical across distros here, unlike chrony, so nothing else needed changing. - The sysctl list is now a variable. It unconditionally set net.ipv4.ip_forward=1, which is correct on the hypervisors but would make edge-1 a router between IoT VLAN 30 and service VLAN 40 once the tagged interface exists, bypassing the UDM's inter-VLAN policy. edge_host keeps only ip_nonlocal_bind; VRRP does not need forwarding. The two roles have different reach by design. chrony runs everywhere: edge-1 serves from its VLAN 30 address (10.2.30.108) and `allow 10.2.0.0/16` plus inter-VLAN routing covers every client, so no network reconfiguration is needed. keepalived is gated on ntp_keepalived_enabled, currently false for edge-1, because VRRP needs layer-2 adjacency on VLAN 40 and the host has no enp1s0.40 yet -- the switch port permits the VLAN but /etc/systemd/network/ has no matching .netdev/.network pair. edge-1's keepalived priority is 70, the lowest, so it stays BACKUP while the PVE nodes exist and wins the VIP by default once they are gone, with no config change at cutover. Verified: playbook syntax-checks; ntp_cluster resolves to the four hosts; Arch paths confirmed against the package file list. Not yet run against any host, and runtime var resolution is unverified (needs the vault password). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Applied to edge-1 and verified. Updates the "not verified" section of the description.
Dry run — jupiter (regression check, the one that mattered):
ok=18 changed=1 failed=0.chrony.conf,keepalived.confand the sysctl entries all report ok, not changed — the refactor is byte-identical on Debian, so no chrony restart on the production NTP servers. The single change is the health-check script, and its diff is comments plus one log-message capitalization; the functional line still renderssystemctl is-active --quiet chronyon Debian. No handler fires on that task, so keepalived is not restarted either. That the live files match the recovered role is also good evidence this really is the role that deployed.Dry run — edge-1:
ok=7 changed=4 failed=1. The failure is a check-mode artifact: the package is not really installed in check mode, so thechronydunit does not exist yet for the enable/start task. Same for thefailed to look up user chronywarnings — the sysusers entry is created by the package install. Both resolve in a real run, confirmed below. Worth noting the dry run cannot fully validate this path on a host where chrony is absent.Applied to edge-1:
ok=14 changed=6 failed=0. All keepalived tasks correctly skipped viantp_keepalived_enabled: false.Verified live on edge-1:
chronydactive and enabled;systemd-timesyncdinactive0.0.0.0:123ptbtime1.ptb.de, stratum 2 (it sits at its ownlocal stratum 10for the first ~minute before converging — worth knowing so a check straight after apply is not misread as a failure)chronyc clientsempty, as expected: DHCP option 42 still points at10.2.40.4Group vars confirmed resolving on the host:
hwts=False,keepalived_enabled=False,iface=enp1s0.40, and sysctl carrying onlyip_nonlocal_bindwithip_forwarddropped.Still outstanding: the PVE nodes have NOT been applied — the only delta there is the cosmetic health-check rewrite, which is not worth touching production NTP for 150 clients on its own; it will ride along with the next real run.
enp1s0.40and enabling keepalived on edge-1 remain a separate step.Unrelated pre-existing bug noticed:
ansible/inventory/api.proxmox.yamlfails to parse on every run (Invalid host pattern 'plugin:'), so the dynamic Proxmox inventory is silently not loading.Prerequisite for enabling keepalived on edge-1: VRRP needs layer-2 adjacency with the other NTP members on the service VLAN, and the host has only untagged VLAN 30. The switch port already permits VLAN 40. Built to be additive and revertible, because enp1s0 is this headless host's only path to the network: - The parent's 20-wired.network is not edited. The VLAN is attached through a 20-wired.network.d/ drop-in instead, so reverting is deleting three paths. - The VLAN .network sorts before 20-wired.network and matches explicitly on Name=. 20-wired.network matches broadly on Type=ether; a VLAN link types as 'vlan' rather than 'ether' (confirmed against networkctl output) so it should not match anyway, but ordering makes that belt-and-braces. - UseRoutes, UseDNS, UseNTP and IPv6AcceptRA are all disabled on the VLAN. Taking a second default route and DNS from the VLAN 40 lease would make return traffic asymmetric across the two VLANs; untagged VLAN 30 stays the only default route. Tasks are tagged vlan40 so they can be applied without running the whole edge-host role, which would restart the Z-Wave, open3e and alloy services. Reconfiguring the parent link is what materialises the VLAN child -- a bare networkctl reload leaves existing links untouched -- so the handler does that explicitly. That re-runs DHCP on enp1s0 and is the one genuinely risky step; worth running with console access to hand. Not yet applied: the apply was blocked pending authorisation, so edge-1 remains in its original network state with only 20-wired.network and 80-can0.network. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Fully applied and verified across all four hosts. Supersedes the "not applied" caveats in the description and the earlier comment.
chronychronychronychronydPVE nodes:
changed=1 failed=0on each, exactly as the dry run predicted — only the health-check script, which has no handler, so neither chrony nor keepalived restarted. The VIP never moved and client service was uninterrupted throughout. Applied to a BACKUP node first, then the remaining two.edge-1:
ok=22 changed=7 failed=0. Joined the VRRP group and transitioned to BACKUP at 21:58:55 per/var/log/keepalived-state.log. It does not hold the VIP, which is correct at priority 70./var/log/keepalived-ntp-health.logis empty — no failed checks. That empty log is the direct evidence for the health-check fix: with the unit name hardcoded to Debian'schrony, every check would fail on Arch and edge-1 would sit in FAULT instead of BACKUP.The last row of the table is the whole point of the PR —
chronyvschronydin the same cluster, handled by the same role.VLAN 40 on edge-1 (second commit) applied cleanly:
enp1s0.40up as typevlanwith lease 10.2.40.153/24. Guards verified — exactly one IPv4 default route (still VLAN 30 via 10.2.30.1), IPv6 default unchanged, no DNS servers on the VLAN link. jupiter and the VIP both answer at ~0.5ms over that interface. An auto-revert timer was armed before the change and disarmed after.edge-1 upstreams confirmed identical to the PVE nodes: PTB stratum-1 (
ptbtime1preferred) plusde.pool.ntp.org. Currently locked to ptbtime1 atReach 377, offset -45µs. Thelocal stratum 10line is what matters for the migration — edge-1 keeps serving from its own clock through a WAN outage, which is why it is a valid time source for Talos where public NTP alone would not be.Ready to merge. Post-migration behaviour needs no further change: when the PVE nodes are rebuilt as bare-metal Talos, edge-1 is the only remaining VRRP member and takes 10.2.40.4 automatically.
Unrelated pre-existing bug, worth a separate issue:
ansible/inventory/api.proxmox.yamlfails to parse on every run (Invalid host pattern 'plugin:'), so the dynamic Proxmox inventory is silently not loading.