fix: [alert] KubeletTooManyPods (cluster) (#937) #938

2026-09-23T19:44:06Z

issue-fix-bot commented

2026-09-23 19:44:06 +00:00

What this PR does

KubeletTooManyPods (info) is firing on k8s-2. That node runs 105 pods against the kubelet default limit of 110. The cause is capacity: about 207 pods on three nodes with 110 slots each leaves no room to lose a node. During a rolling drain, the two remaining nodes go above the 95% threshold. The scheduler also does not move pods back afterwards, so one node stays nearly full after maintenance ends. This PR sets kubelet maxPods: 200 in the shared Talos patch. That fits inside each node's /24 pod CIDR, which Cilium allocates from in ipam.mode: kubernetes. With 3 × 200 slots, about 207 pods fit on two nodes during a drain.

Changes

kubernetes/talos/patches/cluster.yaml — adds maxPods: 200 to machine.kubelet.extraConfig, next to the image-GC settings, with a comment explaining how the value was chosen (IP space per node, N-1 headroom).

Verification

Checked live this run:

./.ci-diag/kubectl get --raw .../api/v1/query?query=ALERTS{alertname="KubeletTooManyPods"} → alertstate="firing", node="k8s-2", instance="10.2.46.12:10250". It is still firing.
.../api/v1/query?query=max by(node)(kubelet_running_pods) → k8s-1 28, k8s-2 105, k8s-3 74, about 207 in total. This matches the issue.
./.ci-diag/kubectl get nodes -o jsonpath=... → all nodes have allocatable pods 110. Pod CIDRs are 10.244.0.0/24, 10.244.4.0/24 and 10.244.2.0/24, so each node has 256 addresses.
kubernetes/infrastructure/cilium/helmrelease.yaml has ipam.mode: kubernetes, so pod IPs come from the node's /24 podCIDR. 200 stays below the roughly 253 usable addresses.

This does not apply through Flux. It is a Talos machine config change. After merging, a human has to regenerate each node's config and apply it one node at a time, following the repo's normal procedure (talosctl -n <node> apply-config --mode=try ..., then make it permanent; see kubernetes/RUNBOOK-cold-rebuild.md). Then check:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.status.allocatable.pods}{"\n"}{end}' shows 200 on every node.
KubeletTooManyPods clears on k8s-2: 105/200 is below the 95% threshold.

Notes

Immediate relief (manual, optional): k8s-2 is still unbalanced (105 pods, while k8s-1 has 28). Until the config is applied, running kubectl rollout restart on a few Deployments on k8s-2 lets the scheduler place their pods on k8s-1. I did not add a descheduler: that would be a new dependency/chart, which is outside this agent's scope. The owner can decide on it separately.
Raising maxPods does not change CPU or memory scheduling; resource requests still limit pod density. It only removes the pod-count limit as the thing that fails first during a drain.
I did not take up the issue's alternative of routing the alert to null. That would silence the alert, and the risk it points at is real: 5 free slots on k8s-2.
The issue's side note about VolumeFailedDelete ... still attached to node k8s-1 events was not investigated here. It is unrelated to this change and should be handled separately.

Closes #937

## What this PR does `KubeletTooManyPods` (info) is firing on k8s-2. That node runs 105 pods against the kubelet default limit of 110. The cause is capacity: about 207 pods on three nodes with 110 slots each leaves no room to lose a node. During a rolling drain, the two remaining nodes go above the 95% threshold. The scheduler also does not move pods back afterwards, so one node stays nearly full after maintenance ends. This PR sets kubelet `maxPods: 200` in the shared Talos patch. That fits inside each node's /24 pod CIDR, which Cilium allocates from in `ipam.mode: kubernetes`. With 3 × 200 slots, about 207 pods fit on two nodes during a drain. ## Changes - `kubernetes/talos/patches/cluster.yaml` — adds `maxPods: 200` to `machine.kubelet.extraConfig`, next to the image-GC settings, with a comment explaining how the value was chosen (IP space per node, N-1 headroom). ## Verification Checked live this run: - `./.ci-diag/kubectl get --raw .../api/v1/query?query=ALERTS{alertname="KubeletTooManyPods"}` → `alertstate="firing", node="k8s-2", instance="10.2.46.12:10250"`. It is still firing. - `.../api/v1/query?query=max by(node)(kubelet_running_pods)` → k8s-1 `28`, k8s-2 `105`, k8s-3 `74`, about 207 in total. This matches the issue. - `./.ci-diag/kubectl get nodes -o jsonpath=...` → all nodes have allocatable pods `110`. Pod CIDRs are `10.244.0.0/24`, `10.244.4.0/24` and `10.244.2.0/24`, so each node has 256 addresses. - `kubernetes/infrastructure/cilium/helmrelease.yaml` has `ipam.mode: kubernetes`, so pod IPs come from the node's /24 podCIDR. 200 stays below the roughly 253 usable addresses. **This does not apply through Flux.** It is a Talos machine config change. After merging, a human has to regenerate each node's config and apply it one node at a time, following the repo's normal procedure (`talosctl -n <node> apply-config --mode=try ...`, then make it permanent; see `kubernetes/RUNBOOK-cold-rebuild.md`). Then check: - `kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.status.allocatable.pods}{"\n"}{end}'` shows `200` on every node. - `KubeletTooManyPods` clears on k8s-2: 105/200 is below the 95% threshold. ## Notes - **Immediate relief (manual, optional):** k8s-2 is still unbalanced (105 pods, while k8s-1 has 28). Until the config is applied, running `kubectl rollout restart` on a few Deployments on k8s-2 lets the scheduler place their pods on k8s-1. I did not add a descheduler: that would be a new dependency/chart, which is outside this agent's scope. The owner can decide on it separately. - Raising `maxPods` does not change CPU or memory scheduling; resource requests still limit pod density. It only removes the pod-count limit as the thing that fails first during a drain. - I did not take up the issue's alternative of routing the alert to `null`. That would silence the alert, and the risk it points at is real: 5 free slots on k8s-2. - The issue's side note about `VolumeFailedDelete ... still attached to node k8s-1` events was not investigated here. It is unrelated to this change and should be handled separately. Closes #937

issue-fix-bot added 1 commit 2026-09-23 19:44:06 +00:00

fix: [alert] KubeletTooManyPods (cluster) (#937 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

68e71d7e52

issue-fix-bot referenced this pull request

2026-09-23 19:44:06 +00:00

[alert] KubeletTooManyPods (cluster) #937

review-bot reviewed

2026-09-23 19:46:12 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 68e71d7e52674bdf9232d65b228c07b8191540a6.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR increases the kubelet maxPods limit from the default 110 to 200 in the Talos cluster configuration to resolve the KubeletTooManyPods alert firing on k8s-2 during node drains. The change is technically sound, well-motivated, and includes thorough documentation of the rationale and verification steps.

🟢 Nits

None. The YAML formatting is correct, the comment explaining the decision is comprehensive, and the placement alongside existing kubelet settings is consistent with the codebase style.

Tests

This is an infrastructure configuration change to the Talos machine config and does not require unit tests. The PR provides clear manual verification steps to confirm the alert clears after applying the configuration:

Check that kubectl get nodes shows 200 allocatable pods per node
Verify KubeletTooManyPods alert clears on k8s-2 when utilization drops below the 95% threshold
Monitor pod density during the next rolling maintenance window

Technical Verification

Confirmed assumptions:

kubernetes/infrastructure/cilium/helmrelease.yaml line 64: ipam.mode: kubernetes is correctly configured ✓
Pod CIDR allocation: 10.244.0.0/16 divided into /24 per node; each /24 has ~256 IPs, leaving ~253 usable after Cilium overhead ✓
Node-specific patches (k8s-1.yaml, site-homelab.yaml) do not override this setting, so it applies uniformly to all three nodes ✓
Math checks: with ~207 pods on 3 nodes, a drain leaves survivors at ~104 pods per node (52% of 200 limit), well below the 95% alert threshold ✓

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `68e71d7e52674bdf9232d65b228c07b8191540a6`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR increases the kubelet `maxPods` limit from the default 110 to 200 in the Talos cluster configuration to resolve the KubeletTooManyPods alert firing on k8s-2 during node drains. The change is technically sound, well-motivated, and includes thorough documentation of the rationale and verification steps. ## 🟢 Nits None. The YAML formatting is correct, the comment explaining the decision is comprehensive, and the placement alongside existing kubelet settings is consistent with the codebase style. ## Tests This is an infrastructure configuration change to the Talos machine config and does not require unit tests. The PR provides clear manual verification steps to confirm the alert clears after applying the configuration: - Check that `kubectl get nodes` shows `200` allocatable pods per node - Verify `KubeletTooManyPods` alert clears on k8s-2 when utilization drops below the 95% threshold - Monitor pod density during the next rolling maintenance window ## Technical Verification Confirmed assumptions: - `kubernetes/infrastructure/cilium/helmrelease.yaml` line 64: `ipam.mode: kubernetes` is correctly configured ✓ - Pod CIDR allocation: `10.244.0.0/16` divided into /24 per node; each /24 has ~256 IPs, leaving ~253 usable after Cilium overhead ✓ - Node-specific patches (k8s-1.yaml, site-homelab.yaml) do not override this setting, so it applies uniformly to all three nodes ✓ - Math checks: with ~207 pods on 3 nodes, a drain leaves survivors at ~104 pods per node (52% of 200 limit), well below the 95% alert threshold ✓ ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner closed this pull request

2026-09-24 05:16:11 +00:00

hermes-bot referenced this pull request

2026-10-21 07:34:20 +00:00

[alert] KubeletTooManyPods (cluster) #937

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#938