docs(runbook): record the half-open socket failure after a node move #630

2026-08-26T10:13:36Z

claude-bot commented

2026-08-26 10:13:36 +00:00

The last open item from the conversion itself.

Anything holding a long-lived TCP session to a device outside the cluster keeps a half-open socket when the node under it moves. The peer drops its side; the pod never learns, because no RST arrives and no keepalive fires.

Why this is worth writing down

The symptom is that there is no symptom. Home Assistant held seven ESTABLISHED sockets to ESPHome devices that were themselves showing "not connected". It logged nothing and retried nothing — as far as it knew, the connections were fine. The search went through logs that were empty by construction.

0B04F40A:E21A  DB1E020A:17A5  01   -> 10.2.30.219  ESTABLISHED
0B04F40A:983C  AA1E020A:17A5  01   -> 10.2.30.170  ESTABLISHED
...
0B04F40A:9DDA  751E020A:17A5  02   -> 10.2.30.117  SYN_SENT

The single 02 was the only device genuinely offline — and the only one Home Assistant did log and retry. That asymmetry is the giveaway: the devices being retried are fine as far as the socket layer is concerned; the ones failing silently are the broken ones.

The check

kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \
  sh -c grep

The last open item from the conversion itself. Anything holding a long-lived TCP session to a device outside the cluster keeps a **half-open socket** when the node under it moves. The peer drops its side; the pod never learns, because no RST arrives and no keepalive fires. ## Why this is worth writing down The symptom is that there is no symptom. Home Assistant held **seven ESTABLISHED sockets** to ESPHome devices that were themselves showing "not connected". It logged nothing and retried nothing — as far as it knew, the connections were fine. The search went through logs that were empty by construction. ``` 0B04F40A:E21A DB1E020A:17A5 01 -> 10.2.30.219 ESTABLISHED 0B04F40A:983C AA1E020A:17A5 01 -> 10.2.30.170 ESTABLISHED ... 0B04F40A:9DDA 751E020A:17A5 02 -> 10.2.30.117 SYN_SENT ``` The single `02` was the only device genuinely offline — and the only one Home Assistant *did* log and retry. That asymmetry is the giveaway: the devices being retried are fine as far as the socket layer is concerned; the ones failing silently are the broken ones. ## The check ```sh kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \ sh -c grep

claude-bot added 1 commit 2026-08-26 10:13:37 +00:00

docs(runbook): record the half-open socket failure after a node move

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

c8b0d1cd4b

Anything holding a long-lived TCP session to a device outside the cluster keeps
a half-open socket when the node under it moves. The peer drops its side; the
pod never learns, because no RST arrives and no keepalive fires.

This cost real time after the conversion. Home Assistant held seven ESTABLISHED
sockets to ESPHome devices that were themselves reporting "not connected". It
logged nothing and retried nothing, because as far as it knew the connections
were fine — so the search went through logs that were empty by construction.
The one device Home Assistant did log was the only one genuinely offline.

Records the check that settles it in one command, reading /proc/net/tcp in the
pod, and notes that reloading the integration is the lighter fix than
restarting the pod.

Placed in "Verify and restore" rather than the known-stale list: it is something
to check before calling a conversion finished, not a trap in the tooling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-26 10:14:58 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit c8b0d1cd4b191da8aab2b8f690f0f4a34bb29072.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR adds a new troubleshooting section to the baremetal conversion runbook documenting a subtle but disruptive failure mode: long-lived TCP sessions to external devices become half-open when cluster nodes migrate, appearing ESTABLISHED locally while the remote device has disconnected. The section provides clear diagnostics and solutions. No blocking issues; the documentation is technically accurate, well-placed, and follows the runbook's style.

🟢 Nits

Line 403: Port hex notation could note that users can use printf '%d\n' 0x17A5 to verify conversions inline, but the comment suffices as-is.

Tests

N/A — documentation only.

Recommendation

Looks reasonable — no blocking issues; comment above is optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `c8b0d1cd4b191da8aab2b8f690f0f4a34bb29072`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR adds a new troubleshooting section to the baremetal conversion runbook documenting a subtle but disruptive failure mode: long-lived TCP sessions to external devices become half-open when cluster nodes migrate, appearing ESTABLISHED locally while the remote device has disconnected. The section provides clear diagnostics and solutions. No blocking issues; the documentation is technically accurate, well-placed, and follows the runbook's style. ## 🟢 Nits - Line 403: Port hex notation could note that users can use `printf '%d\n' 0x17A5` to verify conversions inline, but the comment suffices as-is. ## Tests N/A — documentation only. ## Recommendation Looks reasonable — no blocking issues; comment above is optional.

k.renner merged commit 4187aa91cb into master

2026-08-26 11:04:38 +00:00

k.renner referenced this issue from a commit

2026-08-26 11:04:38 +00:00

Merge pull request 'docs(runbook): record the half-open socket failure after a node move' (#630) from runbook-stale-sockets into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#630