konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

cnpg: cap stopDelay to bound the barman-plugin failover wedge #18

Merged
k.renner merged 1 commits from fix/cnpg-stopdelay-wedge-mitigation into master 2026-06-15 08:57:38 +00:00
Conversation 0 Commits 1 Files Changed 3
+18
k.renner commented 2026-06-15 08:38:14 +00:00
Owner
Copy Link
Copy Source

Adds stopDelay: 60 to ak-pg/pprl-pg/im-pg and smartShutdownTimeout: 10 to pprl-pg/im-pg (ak-pg already had it).

Bounds the post-failover wedge (cloudnative-pg#9770: barman sidecar never terminates -> pod hangs Terminating) from ~30min to ~1min of degraded/down state. smartShutdownTimeout caps the -rw backendless window (validated on ak-pg: 162s -> ~10s).

NOTE: changing stopDelay alters terminationGracePeriodSeconds, so CNPG will roll each cluster's pods once on apply. Recommend merging when it can be watched; validate ak-pg's stopDelay with one failover before relying on it for the others.

Adds stopDelay: 60 to ak-pg/pprl-pg/im-pg and smartShutdownTimeout: 10 to pprl-pg/im-pg (ak-pg already had it). Bounds the post-failover wedge (cloudnative-pg#9770: barman sidecar never terminates -> pod hangs Terminating) from ~30min to ~1min of degraded/down state. smartShutdownTimeout caps the -rw backendless window (validated on ak-pg: 162s -> ~10s). NOTE: changing stopDelay alters terminationGracePeriodSeconds, so CNPG will roll each cluster's pods once on apply. Recommend merging when it can be watched; validate ak-pg's stopDelay with one failover before relying on it for the others.
k.renner added 1 commit 2026-06-15 08:38:15 +00:00
cnpg: cap stopDelay (+smartShutdownTimeout) to bound failover wedge 3632640b92 
Roll out shutdown tuning across the barman-backed CNPG clusters:

- stopDelay: 60 (all three) — caps terminationGracePeriodSeconds. With the
  barman-cloud plugin the demoted/restarting pod's instance-manager deadlocks
  on shutdown: a plugin runnable misses the 30s controller-runtime grace, the
  process never exits, and the native sidecar (which only stops after the main
  container) keeps running -> pod hangs Terminating (upstream
  cloudnative-pg#9770). Default 1800s leaves 2-instance clusters at 1/2 (and
  single-instance im-pg fully down) for ~30min after each failover/restart;
  60s caps that to ~1min. Postgres itself is cleanly down by ~12s, so the
  trimmed time is only the wedged instance-manager shell.
- smartShutdownTimeout: 10 (pprl-pg, im-pg) — already on ak-pg (#16); caps the
  smart-shutdown drain so -rw isn't backendless for 180s. Validated on ak-pg:
  app downtime 162s -> ~10s.

Note: changing stopDelay alters terminationGracePeriodSeconds, so CNPG rolls
each cluster's pods once to apply it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.renner merged commit a3a678ba34 into master 2026-06-15 08:57:38 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#18