talos: lower kubelet image GC thresholds to 70/60 #13

2026-06-12T21:23:19Z

k.renner commented

2026-06-12 21:23:19 +00:00

Summary

Lowers kubelet image GC thresholds from the upstream defaults (85% high / 80% low) to 70/60 so containerd's image cache is pruned before /var crosses the line where Ceph fires MON_DISK_LOW on the CP VMs.

Why

Node	`/var` used	Pods scheduled	Mon
k8s-1	59%	42	c (ok)
k8s-2	79%	75	b (warning)
k8s-3	77%	~mid	a (warning)
k8s-edge	67%	14	—

The mon DB itself is 118 MiB on each mon — Ceph's MON_DISK_LOW is firing on the host filesystem under the mon's hostpath (/var/lib/rook), not on the DB.
/var is shared with containerd's image store + /var/log/pods + ephemeral. k8s-2/k8s-3 carry ~80% more pods than k8s-1 (heavy ML pods on k8s-2: whisper, openwakeword, influxdb-influxdb2, multiple authentik replicas), so they accumulate more image layers.
Growth rate is ~0.5 GiB/day across all nodes over the last 7d — steady accumulation, no acute leak. Kubelet's default 85% high watermark sits above where we land, so GC never fires and unused images pile up indefinitely.

Patch

machine:
  kubelet:
    extraConfig:
      imageGCHighThresholdPercent: 70
      imageGCLowThresholdPercent: 60

Lands in kubernetes/talos/patches/cluster.yaml (cluster-wide patch, applies to all nodes — CPs and edge).

Why these numbers

70% high keeps /var always under 30%-free, never crossing mon_data_avail_warn (default 30).
60% low gives kubelet 10% of headroom to GC into before the next pull starts refilling.
Spegel P2P pull-through cache softens the re-pull cost when a GC'd image is needed back — first node hits docker.io/quay.io, every subsequent node peer-fetches.

Rollout

Per Talos v1.12 docs, extraConfig changes require a node reboot. Standing discipline (per CLAUDE.md):

kubectl drain first (PDBs respected; never skip drain even if eviction blocks)
Wait for CephCluster HEALTH_OK between nodes (not just node Ready — Ceph quorum is slower than kubelet)
One CP at a time

Suggest applying to k8s-1 first (lowest pressure, mon c not warning, safest single-node validation), then k8s-3, then k8s-2. k8s-edge can be rolled at the end (or skipped this cycle — not currently warning).

Choice of pipeline (deploy-infrastructure vs other) is the operator's call — pipeline is not specified here.

Test plan

Apply to k8s-1; node returns Ready; Ceph HEALTH_OK
talosctl -n k8s-1 read /etc/kubernetes/kubelet.yaml reflects 70/60 (or equivalent kubelet config dump)
Apply to k8s-3, wait for Ceph HEALTH_OK
Apply to k8s-2, wait for Ceph HEALTH_OK
After 24h: confirm node_filesystem_avail_bytes{mountpoint="/var"} on k8s-2/k8s-3 shows GC activity (used% stabilizes at ≤70% rather than continuing to climb)
MON_DISK_LOW warning clears

🤖 Generated with Claude Code

## Summary Lowers kubelet image GC thresholds from the upstream defaults (85% high / 80% low) to **70/60** so containerd's image cache is pruned before `/var` crosses the line where Ceph fires `MON_DISK_LOW` on the CP VMs. ## Why | Node | `/var` used | Pods scheduled | Mon | |---|---|---|---| | k8s-1 | 59% | 42 | c (ok) | | k8s-2 | 79% | 75 | b (**warning**) | | k8s-3 | 77% | ~mid | a (**warning**) | | k8s-edge | 67% | 14 | — | - The mon DB itself is **118 MiB** on each mon — Ceph's `MON_DISK_LOW` is firing on the host filesystem under the mon's hostpath (`/var/lib/rook`), not on the DB. - `/var` is shared with containerd's image store + `/var/log/pods` + ephemeral. k8s-2/k8s-3 carry ~80% more pods than k8s-1 (heavy ML pods on k8s-2: whisper, openwakeword, influxdb-influxdb2, multiple authentik replicas), so they accumulate more image layers. - Growth rate is **~0.5 GiB/day across all nodes** over the last 7d — steady accumulation, no acute leak. Kubelet's default 85% high watermark sits above where we land, so GC never fires and unused images pile up indefinitely. ## Patch ```yaml machine: kubelet: extraConfig: imageGCHighThresholdPercent: 70 imageGCLowThresholdPercent: 60 ``` Lands in `kubernetes/talos/patches/cluster.yaml` (cluster-wide patch, applies to all nodes — CPs and edge). ## Why these numbers - 70% high keeps `/var` always under 30%-free, never crossing `mon_data_avail_warn` (default 30). - 60% low gives kubelet 10% of headroom to GC into before the next pull starts refilling. - Spegel P2P pull-through cache softens the re-pull cost when a GC'd image is needed back — first node hits docker.io/quay.io, every subsequent node peer-fetches. ## Rollout Per Talos v1.12 docs, `extraConfig` changes **require a node reboot**. Standing discipline (per CLAUDE.md): - `kubectl drain` first (PDBs respected; never skip drain even if eviction blocks) - Wait for CephCluster `HEALTH_OK` between nodes (not just node Ready — Ceph quorum is slower than kubelet) - One CP at a time Suggest applying to k8s-1 first (lowest pressure, mon c not warning, safest single-node validation), then k8s-3, then k8s-2. k8s-edge can be rolled at the end (or skipped this cycle — not currently warning). Choice of pipeline (`deploy-infrastructure` vs other) is the operator's call — pipeline is not specified here. ## Test plan - [ ] Apply to k8s-1; node returns Ready; Ceph `HEALTH_OK` - [ ] `talosctl -n k8s-1 read /etc/kubernetes/kubelet.yaml` reflects 70/60 (or equivalent kubelet config dump) - [ ] Apply to k8s-3, wait for Ceph `HEALTH_OK` - [ ] Apply to k8s-2, wait for Ceph `HEALTH_OK` - [ ] After 24h: confirm `node_filesystem_avail_bytes{mountpoint="/var"}` on k8s-2/k8s-3 shows GC activity (used% stabilizes at ≤70% rather than continuing to climb) - [ ] `MON_DISK_LOW` warning clears 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-12 21:23:19 +00:00

talos: lower kubelet image GC thresholds to 70/60 1fcda644c0

The CP VMs are seeing Ceph `MON_DISK_LOW` on mons a,b because their
shared /var (xfs on /dev/vda5) sits at 77-79% used — past Ceph's
default `mon_data_avail_warn` line of <30% free. Mon DB itself is
118 MiB; the disk pressure is pod density (k8s-2 hosts 75 pods vs
k8s-1's 42), driving containerd image cache and /var/log/pods.

Kubelet's default image GC thresholds (85% high / 80% low) sit
above where we land, so GC never fires and unused image layers
accumulate indefinitely. Lowering to 70/60 forces eviction before
/var crosses the mon warn line, with Spegel's P2P pull-through cache
softening the re-pull cost on the rare case a GC'd image is needed
back.

Requires a node reboot to take effect (Talos `extraConfig` change).
Roll one CP at a time with `kubectl drain` + Ceph `HEALTH_OK` wait
between nodes, per the standing drain/Ceph-wait discipline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

k.renner merged commit ad17463e63 into master

2026-06-13 07:49:05 +00:00

k.renner referenced this issue from a commit

2026-06-13 07:49:06 +00:00

Merge pull request 'talos: lower kubelet image GC thresholds to 70/60' (#13) from fix/kubelet-image-gc-thresholds into master

claude-bot referenced this pull request

2026-10-25 20:06:12 +00:00

feat(mqtt): admin login as a register account #1437

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#13