otbr: make the probes watch Thread, not just the process #141

2026-07-09T18:24:14Z

k.renner commented

2026-07-09 18:24:14 +00:00

Why

The only probe on this container was pgrep -x otbr-agent. It exists because the upstream entrypoint keeps PID 1 alive with tail -f after the agent dies. It cannot catch what actually happened:

On 2026-06-27 otbr-agent ran for twelve days with ot-ctl state = disabled and the border router stopped, while the pod reported Running 1/1 the entire time.

A live process is not a live Thread mesh.

What

probe	checks	on failure
startupProbe (new)	`ot-ctl`'s control socket answers	10s × 60 = 10 min, then restart
readinessProbe (new)	state ∈ `leader\|router\|child\|detached`	endpoints removed
livenessProbe (extended)	`pgrep`, plus restart when state is `disabled` and a dataset is persisted	restart

detached counts as ready: the stack is up and attaching, REST still serves, and treating normal mesh churn as NotReady would flap HA's OTBR integration for nothing. disabled does not count — it is the twelve-day failure, and the one state that never resolves on its own.

The two guards that keep this from being worse than the bug

The dataset guard on liveness is load-bearing. On a clean install there is no dataset and disabled is correct until otbr-dataset-bootstrap seeds one over kubectl exec. Restarting there would fight the Job — which the wiki records dying with BackoffLimitExceeded against a CrashLoopBackOff. So liveness only restarts on disabled with a dataset, i.e. activation that should have happened and didn't.

A wedged control socket must not cause a restart storm. If ot-ctl doesn't answer, $S is empty, the state check falls through, and liveness rests on pgrep alone. The pod goes NotReady (visible) without being killed.

All three probes strip CR — ot-ctl terminates lines with CRLF, so ot-ctl state is leader\r, matching no shell glob (the bug from #140). curl/wget are absent from this image, so probes go through ot-ctl, not REST.

Removing endpoints only affects HA's OTBR REST integration on :8081. matter-server does not consume this Service — it learns the OMR prefix by RA on its own macvlan interface.

Verified

Scripts run against the live pod (state=leader): startup, readiness, liveness all exit 0.

States that can't be produced on a healthy border router were driven with stubbed ot-ctl/pgrep (the stub emits CRLF, like the real thing):

  state       dataset  agent  | startup readiness liveness
  leader      yes      alive  |    0        0         0
  router      yes      alive  |    0        0         0
  detached    yes      alive  |    0        0         0     <- no flap
  disabled    yes      alive  |    0        1         1     <- THE BUG: restart
  disabled    no       alive  |    0        1         0     <- clean install: no restart
  <wedged>    yes      alive  |    1        1         0     <- NotReady, no storm
  leader      yes      dead   |    0        0         1

sh -n clean on both probe scripts as rendered by kubectl kustomize; kustomize builds; the entrypoint is untouched.

Caveat

An operator running ot-ctl thread stop by hand on a provisioned node gets the container restarted ~3 minutes later. That is deliberate: disabled with a dataset is a fault on this deployment, not a maintenance state. If you want a maintenance mode, the honest way is to scale the Deployment to 0, not to leave a border router pretending to be up.

Merging is the test

Expect the pod to roll, startupProbe to pass once ot-ctl answers, Thread attached as leader, then 1/1 Ready. I'll verify kubectl get pod shows Ready only after Thread attaches — which is the whole point.

## Why The only probe on this container was `pgrep -x otbr-agent`. It exists because the upstream entrypoint keeps PID 1 alive with `tail -f` after the agent dies. It cannot catch what actually happened: > On 2026-06-27 `otbr-agent` ran for **twelve days** with `ot-ctl state` = `disabled` and the border router `stopped`, while the pod reported `Running 1/1` the entire time. A live process is not a live Thread mesh. ## What | probe | checks | on failure | |---|---|---| | **startupProbe** *(new)* | `ot-ctl`'s control socket answers | 10s × 60 = 10 min, then restart | | **readinessProbe** *(new)* | state ∈ `leader\|router\|child\|detached` | endpoints removed | | **livenessProbe** *(extended)* | `pgrep`, **plus** restart when state is `disabled` **and** a dataset is persisted | restart | `detached` counts as **ready**: the stack is up and attaching, REST still serves, and treating normal mesh churn as NotReady would flap HA's OTBR integration for nothing. `disabled` does not count — it is the twelve-day failure, and the one state that never resolves on its own. ## The two guards that keep this from being worse than the bug **The dataset guard on liveness is load-bearing.** On a clean install there is no dataset and `disabled` is *correct* until `otbr-dataset-bootstrap` seeds one over `kubectl exec`. Restarting there would fight the Job — which the wiki records dying with `BackoffLimitExceeded` against a `CrashLoopBackOff`. So liveness only restarts on `disabled` **with** a dataset, i.e. activation that should have happened and didn't. **A wedged control socket must not cause a restart storm.** If `ot-ctl` doesn't answer, `$S` is empty, the state check falls through, and liveness rests on `pgrep` alone. The pod goes NotReady (visible) without being killed. All three probes strip CR — `ot-ctl` terminates lines with CRLF, so `ot-ctl state` is `leader\r`, matching no shell glob (the bug from #140). `curl`/`wget` are absent from this image, so probes go through `ot-ctl`, not REST. Removing endpoints only affects HA's OTBR REST integration on `:8081`. `matter-server` does **not** consume this Service — it learns the OMR prefix by RA on its own macvlan interface. ## Verified Scripts run against the **live pod** (`state=leader`): startup, readiness, liveness all exit `0`. States that can't be produced on a healthy border router were driven with stubbed `ot-ctl`/`pgrep` (the stub emits CRLF, like the real thing): ``` state dataset agent | startup readiness liveness leader yes alive | 0 0 0 router yes alive | 0 0 0 detached yes alive | 0 0 0 <- no flap disabled yes alive | 0 1 1 <- THE BUG: restart disabled no alive | 0 1 0 <- clean install: no restart <wedged> yes alive | 1 1 0 <- NotReady, no storm leader yes dead | 0 0 1 ``` `sh -n` clean on both probe scripts as rendered by `kubectl kustomize`; kustomize builds; the entrypoint is untouched. ## Caveat An operator running `ot-ctl thread stop` by hand on a provisioned node gets the container restarted ~3 minutes later. That is deliberate: `disabled` with a dataset is a fault on this deployment, not a maintenance state. If you want a maintenance mode, the honest way is to scale the Deployment to 0, not to leave a border router pretending to be up. ## Merging is the test Expect the pod to roll, `startupProbe` to pass once `ot-ctl` answers, `Thread attached as leader`, then `1/1 Ready`. I'll verify `kubectl get pod` shows Ready only *after* Thread attaches — which is the whole point.

k.renner added 1 commit 2026-07-09 18:24:14 +00:00

otbr: make the probes watch Thread, not just the process e174e1d227

The only probe on this container was `pgrep -x otbr-agent`. It caught the
upstream entrypoint's habit of keeping PID 1 alive via `tail -f` after the
agent dies. It could not catch what actually happened on 2026-06-27:
otbr-agent ran for twelve days with `ot-ctl state` = disabled and the
border router stopped, while the pod reported Running 1/1 the whole time.
A live process is not a live Thread mesh.

Add a startupProbe and a readinessProbe, and teach liveness the one
protocol state a restart repairs.

  startupProbe   ot-ctl's control socket answers. 10s x 60 = 10 minutes,
                 covering the apt install, the socat TCP<->PTY bridge to
                 the SLZB radio, and the spinel handshake. While it runs
                 the other probes are suspended, so the entrypoint gets to
                 finish activating Thread before liveness may act.

  readinessProbe state is leader|router|child|detached. `detached` counts
                 as ready: the stack is up and attaching, REST still
                 serves, and treating normal mesh churn as NotReady would
                 flap HA's OTBR integration for nothing. `disabled` does
                 not count -- it is the twelve-day failure and the one
                 state that never resolves on its own.

  livenessProbe  pgrep, plus: restart when state is `disabled` AND a
                 dataset is persisted. That combination means activation
                 never ran, and since #139 a restart re-runs a path that
                 polls for the dataset and starts Thread. Self-healing.

The dataset guard on liveness is load-bearing. On a clean install there is
no dataset and `disabled` is correct until otbr-dataset-bootstrap seeds one
over `kubectl exec`. Restarting there would fight the Job, which has
already been seen to die with BackoffLimitExceeded against a
CrashLoopBackOff. Likewise, if ot-ctl does not answer, the state check
falls through and liveness rests on pgrep alone -- a wedged control socket
must not trigger a restart storm.

All three probes strip CR. ot-ctl speaks a serial-console protocol and
terminates lines with CRLF, so `ot-ctl state` yields `leader\r`, which
matches no shell glob (#140). curl and wget are absent from this image, so
the probes go through ot-ctl rather than REST.

Removing endpoints affects only HA's OTBR REST integration on :8081.
matter-server does not consume this Service -- it learns the OMR prefix by
RA on its own macvlan interface.

Verified. The three scripts were run against the live pod (state=leader):
startup/readiness/liveness all exit 0. The states that cannot be produced
on a healthy border router were driven with stubbed ot-ctl/pgrep:

  state       dataset  agent  | startup readiness liveness
  leader      yes      alive  |    0        0         0
  router      yes      alive  |    0        0         0
  detached    yes      alive  |    0        0         0
  disabled    yes      alive  |    0        1         1   <- the bug
  disabled    no       alive  |    0        1         0   <- clean install
  <wedged>    yes      alive  |    1        1         0
  leader      yes      dead   |    0        0         1

`sh -n` clean on both probe scripts as rendered by kustomize; kustomize
builds. Entrypoint untouched.

Caveat: an operator running `ot-ctl thread stop` by hand on a provisioned
node gets the container restarted ~3 minutes later. Deliberate --
`disabled` with a dataset is a fault on this deployment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit d83bd5aaa4 into master

2026-07-09 18:57:04 +00:00

k.renner referenced this issue from a commit

2026-07-09 18:57:04 +00:00

Merge pull request 'otbr: make the probes watch Thread, not just the process' (#141) from fix/otbr-protocol-aware-probes into master

k.renner referenced this issue from a commit

2026-07-10 14:43:53 +00:00

otbr: bind the REST API to the pod address, not 0.0.0.0

k.renner referenced this pull request

2026-07-10 14:44:13 +00:00

otbr: bind the REST API to the pod address, not 0.0.0.0 #145

Sign in to join this conversation.