fix(chrony): step the clock at boot so a bad RTC self-corrects #465

2026-08-05T14:35:44Z

claude-bot commented

2026-08-05 14:35:44 +00:00

Why

chrony.conf.j2 has no makestep directive, so chronyd only ever slews. A large offset is therefore never closed in practice.

This caused an ~18h outage of all heating telemetry on 2026-08-04/05.

edge-1's CMOS battery is dead. After a power loss the RTC came up reading 2018-08-20, and the clock was then reconstructed entirely in software:

PM: RTC time: 09:18:56, date: 2018-08-20
rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC
systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC

It settled at ~2026-07-19, roughly 16 days behind. chronyd then synced to ptbtime1.ptb.de at stratum 2 with a sub-millisecond RMS offset — and still reported:

System time     : 1376235.375000000 seconds slow of NTP time
Last offset     : -0.000130062 seconds

It knew the correct time and corrected none of it.

The 16 days is incidental — it is just where the software flooring happened to land. There is no 16-day mechanism. The fault this PR addresses is that nothing steps.

With host time before the cluster certificate's NotBefore, every outbound TLS handshake failed x509 validation:

tls: failed to verify certificate: x509: certificate has expired or is not yet
valid: current time 2026-07-20T16:15:36+02:00 is before 2026-07-23T06:15:01Z

Two things broke at once:

The local mosquitto → cluster MQTT bridge (:8883, bridge_insecure false) could not connect, so open3e / e3oncan data never left the host. Every Vitocal 250-A, Vitocharge VX3 and E380 entity in Home Assistant went unavailable. Both services were running and reading CAN fine the whole time.
The Alloy → Loki shipper failed identically, so edge-1 vanished from Loki at the same moment.

Point 2 is the nasty part: the skew that breaks the service also breaks the evidence, so this failure mode is silent to log-based alerting by construction.

What this changes

Adds a makestep directive, parameterised via chrony_makestep_threshold (1.0s) and chrony_makestep_limit (3).

The limit confines steps to the first few updates after chronyd starts. Boot-time corrections jump; a long-running NTP server is never stepped out from under its workloads mid-flight. This is the behaviour most distributions ship by default.

Verification

The live outage was recovered by hand with chronyc makestep, which brought the offset to 0.000000000 seconds immediately and restored both the MQTT bridge and log shipping within a minute. This PR makes that correction happen automatically at boot instead of requiring manual intervention.

Applies to every host using the role, including the PVE nodes — all of them currently share the same never-step behaviour.

Not fixed here

edge-1's dead RTC battery. That is hardware; this PR only ensures the software recovers from it instead of sitting broken. Without the battery replacement edge-1 will still come up with a wrong clock after every power loss — it will just self-correct now.
local stratum {{ chrony_local_stratum }} means these hosts serve NTP even when their own clock is wrong, so edge-1 may have been distributing a ~16-day-stale time to any client pointed at it. Real and separate; changing it touches the NTP HA design, so it is left out rather than bundled in.

Review note

An earlier revision of this PR and its commit message claimed edge-1 "rebooted with an RTC ~16 days in the past". That was wrong — the RTC was at 2018, and the ~16 days came from systemd's epoch flooring. Corrected throughout.

🤖 Generated with Claude Code

## Why `chrony.conf.j2` has no `makestep` directive, so chronyd only ever **slews**. A large offset is therefore never closed in practice. This caused an ~18h outage of all heating telemetry on 2026-08-04/05. **edge-1's CMOS battery is dead.** After a power loss the RTC came up reading **2018-08-20**, and the clock was then reconstructed entirely in software: ``` PM: RTC time: 09:18:56, date: 2018-08-20 rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC ``` It settled at ~2026-07-19, roughly 16 days behind. chronyd then synced to `ptbtime1.ptb.de` at stratum 2 with a sub-millisecond RMS offset — and still reported: ``` System time : 1376235.375000000 seconds slow of NTP time Last offset : -0.000130062 seconds ``` It knew the correct time and corrected none of it. The 16 days is **incidental** — it is just where the software flooring happened to land. There is no 16-day mechanism. The fault this PR addresses is that nothing steps. With host time before the cluster certificate's `NotBefore`, every outbound TLS handshake failed x509 validation: ``` tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-07-20T16:15:36+02:00 is before 2026-07-23T06:15:01Z ``` Two things broke at once: 1. The local mosquitto → cluster MQTT bridge (`:8883`, `bridge_insecure false`) could not connect, so `open3e` / `e3oncan` data never left the host. Every Vitocal 250-A, Vitocharge VX3 and E380 entity in Home Assistant went `unavailable`. Both services were running and reading CAN fine the whole time. 2. The Alloy → Loki shipper failed identically, so `edge-1` vanished from Loki at the same moment. Point 2 is the nasty part: **the skew that breaks the service also breaks the evidence**, so this failure mode is silent to log-based alerting by construction. ## What this changes Adds a `makestep` directive, parameterised via `chrony_makestep_threshold` (1.0s) and `chrony_makestep_limit` (3). The limit confines steps to the first few updates after chronyd starts. Boot-time corrections jump; a long-running NTP server is never stepped out from under its workloads mid-flight. This is the behaviour most distributions ship by default. ## Verification The live outage was recovered by hand with `chronyc makestep`, which brought the offset to `0.000000000 seconds` immediately and restored both the MQTT bridge and log shipping within a minute. This PR makes that correction happen automatically at boot instead of requiring manual intervention. Applies to every host using the role, including the PVE nodes — all of them currently share the same never-step behaviour. ## Not fixed here - **edge-1's dead RTC battery.** That is hardware; this PR only ensures the software recovers from it instead of sitting broken. Without the battery replacement edge-1 will still come up with a wrong clock after every power loss — it will just self-correct now. - **`local stratum {{ chrony_local_stratum }}`** means these hosts serve NTP **even when their own clock is wrong**, so edge-1 may have been distributing a ~16-day-stale time to any client pointed at it. Real and separate; changing it touches the NTP HA design, so it is left out rather than bundled in. ## Review note An earlier revision of this PR and its commit message claimed edge-1 "rebooted with an RTC ~16 days in the past". That was wrong — the RTC was at 2018, and the ~16 days came from systemd's epoch flooring. Corrected throughout. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-05 14:35:44 +00:00

fix(chrony): step the clock at boot so a bad RTC self-corrects

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

7fcaa4b035

chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.

On 2026-08-04 edge-1 rebooted with its RTC ~16 days in the past. chronyd
was synchronised to ptbtime1 at stratum 2 with a sub-millisecond RMS
offset, yet reported "System time: 1376235.375 seconds slow of NTP time"
and corrected none of it. With host time at 2026-07-20, every outbound
TLS handshake failed x509 NotBefore validation against a cluster
certificate issued 2026-07-23:

  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-07-20T16:15:36+02:00 is before
  2026-07-23T06:15:01Z

That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.

Recovery required a manual `chronyc makestep`. Adding the directive lets
this correct itself at boot instead.

The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-05 14:38:22 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 7fcaa4b035076a2ed787fefc97387165222565d6.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a makestep 1.0 3 directive to the chrony-ntp-server role's rendered chrony.conf, parameterised via two new role defaults (chrony_makestep_threshold, chrony_makestep_limit). This closes the gap that caused the 2026-08-04/05 edge-1 outage: without makestep, chronyd only ever slews, so a large boot-time offset (here, a 16-day-stale RTC) is never actually corrected even while chronyd reports itself synchronised. The change is small, well-scoped, and matches the values most distributions ship by default. No blocking issues found.

🟡 Suggestions

ansible/roles/chrony-ntp-server/defaults/main.yml:21-29 and templates/chrony.conf.j2:17-19 carry near-duplicate multi-line explanations of the same rationale. Not wrong, just a bit of upkeep burden if the reasoning ever needs to be revised (two places to update in sync). Consider trimming one to a one-line pointer to the other.
The PR body itself flags that local stratum {{ chrony_local_stratum }} (chrony.conf.j2:29) lets these hosts continue serving NTP to clients even with a wrong clock, which was likely a contributing factor in the actual incident (edge-1 potentially handing out 16-day-stale time before it caught up). The author explicitly and reasonably scopes this out as a separate, larger design change — flagging here only so it doesn't get lost; worth its own follow-up issue/PR.

Tests

No automated tests exist for Ansible roles in this repo (no molecule/lint harness under ansible/roles/chrony-ntp-server), consistent with the rest of the roles directory, so the lack of one here isn't a gap specific to this PR. The change is a straightforward template/defaults addition using standard chrony.conf makestep <threshold> <limit> syntax, and the PR body documents that the equivalent manual command (chronyc makestep) was verified live against the actual incident, immediately closing the offset and restoring the MQTT bridge and log shipping. That's reasonable verification for a two-line config change of this kind.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `7fcaa4b035076a2ed787fefc97387165222565d6`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a `makestep 1.0 3` directive to the `chrony-ntp-server` role's rendered `chrony.conf`, parameterised via two new role defaults (`chrony_makestep_threshold`, `chrony_makestep_limit`). This closes the gap that caused the 2026-08-04/05 edge-1 outage: without `makestep`, chronyd only ever slews, so a large boot-time offset (here, a 16-day-stale RTC) is never actually corrected even while chronyd reports itself synchronised. The change is small, well-scoped, and matches the values most distributions ship by default. No blocking issues found. ## 🟡 Suggestions - `ansible/roles/chrony-ntp-server/defaults/main.yml:21-29` and `templates/chrony.conf.j2:17-19` carry near-duplicate multi-line explanations of the same rationale. Not wrong, just a bit of upkeep burden if the reasoning ever needs to be revised (two places to update in sync). Consider trimming one to a one-line pointer to the other. - The PR body itself flags that `local stratum {{ chrony_local_stratum }}` (chrony.conf.j2:29) lets these hosts continue serving NTP to clients even with a wrong clock, which was likely a contributing factor in the actual incident (edge-1 potentially handing out 16-day-stale time before it caught up). The author explicitly and reasonably scopes this out as a separate, larger design change — flagging here only so it doesn't get lost; worth its own follow-up issue/PR. ## Tests No automated tests exist for Ansible roles in this repo (no molecule/lint harness under `ansible/roles/chrony-ntp-server`), consistent with the rest of the roles directory, so the lack of one here isn't a gap specific to this PR. The change is a straightforward template/defaults addition using standard `chrony.conf` `makestep <threshold> <limit>` syntax, and the PR body documents that the equivalent manual command (`chronyc makestep`) was verified live against the actual incident, immediately closing the offset and restoring the MQTT bridge and log shipping. That's reasonable verification for a two-line config change of this kind. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit d41f4870ad into master

2026-08-05 15:14:22 +00:00

k.renner referenced this issue from a commit

2026-08-05 15:14:23 +00:00

Merge pull request 'fix(chrony): step the clock at boot so a bad RTC self-corrects' (#465) from worktree-chrony-makestep into master

claude-bot referenced this issue from a commit

2026-08-05 15:36:40 +00:00

docs(chrony): correct the root cause recorded for the makestep fix

claude-bot referenced this pull request

2026-08-05 15:36:59 +00:00

docs(chrony): correct the root cause recorded for the makestep fix #466

claude-bot referenced this pull request