konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

authentik: bump memory limits to 2Gi, lift requests #14

Merged
k.renner merged 1 commits from fix/authentik-memory-bump into master 2026-06-13 08:21:31 +00:00
Conversation 0 Commits 1 Files Changed 2
+6 -6
k.renner commented 2026-06-13 08:11:25 +00:00
Owner
Copy Link
Copy Source

Summary

Triples the memory limit for all authentik components from 1Gi → 2Gi and bumps requests to match steady-state. Authentik has been running at 70–80% of its 1Gi cap with no headroom for spikes.

Live numbers (just now)

Pod Mem usage Limit Util %
ak-pg-2 (primary) 577 MiB 1 GiB 56%
authentik-server (×2 mature) 720, 745 MiB 1 GiB 73, 74%
authentik-worker (×2 mature) 380–513 MiB 1 GiB 38–50%

Why

During today's kubelet GC threshold rollout on k8s-2, ak-pg-1 (the postgres primary at the time) failed its readiness probe and CNPG initiated a switchover to ak-pg-2. The switchover completed and authentik recovered via ak-pg-2 in <2 min, but the trigger was a memory-pressure window during the brief kubelet restart — postgres has no slack at 1Gi when checkpoint, vacuum, or a connection spike hits.

authentik-server in particular runs gunicorn workers + Django + policy/blueprint evaluation. A single LDAP sync against the synology/nas-1 outpost is enough to push it past 1Gi.

Changes

File Component req before → after limit before → after
kubernetes/seed/authentik/cluster.yaml ak-pg 512Mi → 768Mi 1Gi → 2Gi
kubernetes/apps/authentik/helmrelease.yaml authentik-server 768Mi → 1Gi 1Gi → 2Gi
kubernetes/apps/authentik/helmrelease.yaml authentik-worker 768Mi → 1Gi 1Gi → 2Gi

Scheduling impact

Additional cluster-wide memory request:

  • ak-pg: +512 MiB (2 instances × 256 MiB)
  • authentik-server: +512 MiB to +1 GiB (2-4 replicas × 256 MiB via HPA)
  • authentik-worker: +512 MiB to +1 GiB (2-4 replicas × 256 MiB via HPA)

Total worst case: ~2.5 GiB across all 4 nodes. CP nodes currently have 13–25 GiB /var free; mem allocation isn't the bound here. Safe.

Rollout

On merge, Flux applies and triggers a rolling restart of authentik-server, authentik-worker, and the ak-pg pods (CNPG handles pg switchover gracefully). Brief blip on ak-pg-rw during pg restart; authentik retries postgres connections on backoff.

Test plan

  • Merge
  • HelmRelease reconciles cleanly
  • ak-pg Cluster reaches healthy state after CNPG-managed restart
  • kubectl -n authentik top pods shows new limits taking effect
  • Authentik UI/LDAP/RADIUS still serving

🤖 Generated with Claude Code

## Summary Triples the memory limit for all authentik components from 1Gi → 2Gi and bumps requests to match steady-state. Authentik has been running at 70–80% of its 1Gi cap with no headroom for spikes. ## Live numbers (just now) | Pod | Mem usage | Limit | Util % | |---|---|---|---| | ak-pg-2 (primary) | 577 MiB | 1 GiB | 56% | | authentik-server (×2 mature) | 720, 745 MiB | 1 GiB | 73, 74% | | authentik-worker (×2 mature) | 380–513 MiB | 1 GiB | 38–50% | ## Why During today's kubelet GC threshold rollout on k8s-2, ak-pg-1 (the postgres primary at the time) failed its readiness probe and CNPG initiated a switchover to ak-pg-2. The switchover completed and authentik recovered via ak-pg-2 in <2 min, but the trigger was a memory-pressure window during the brief kubelet restart — postgres has no slack at 1Gi when checkpoint, vacuum, or a connection spike hits. authentik-server in particular runs gunicorn workers + Django + policy/blueprint evaluation. A single LDAP sync against the synology/nas-1 outpost is enough to push it past 1Gi. ## Changes | File | Component | req before → after | limit before → after | |---|---|---|---| | `kubernetes/seed/authentik/cluster.yaml` | ak-pg | 512Mi → 768Mi | 1Gi → 2Gi | | `kubernetes/apps/authentik/helmrelease.yaml` | authentik-server | 768Mi → 1Gi | 1Gi → 2Gi | | `kubernetes/apps/authentik/helmrelease.yaml` | authentik-worker | 768Mi → 1Gi | 1Gi → 2Gi | ## Scheduling impact Additional cluster-wide memory request: - ak-pg: +512 MiB (2 instances × 256 MiB) - authentik-server: +512 MiB to +1 GiB (2-4 replicas × 256 MiB via HPA) - authentik-worker: +512 MiB to +1 GiB (2-4 replicas × 256 MiB via HPA) Total worst case: ~2.5 GiB across all 4 nodes. CP nodes currently have 13–25 GiB `/var` free; mem allocation isn't the bound here. Safe. ## Rollout On merge, Flux applies and triggers a rolling restart of authentik-server, authentik-worker, and the ak-pg pods (CNPG handles pg switchover gracefully). Brief blip on ak-pg-rw during pg restart; authentik retries postgres connections on backoff. ## Test plan - [ ] Merge - [ ] HelmRelease reconciles cleanly - [ ] ak-pg Cluster reaches healthy state after CNPG-managed restart - [ ] `kubectl -n authentik top pods` shows new limits taking effect - [ ] Authentik UI/LDAP/RADIUS still serving 🤖 Generated with [Claude Code](https://claude.com/claude-code)
k.renner added 1 commit 2026-06-13 08:11:26 +00:00
authentik: bump memory limits to 2Gi, lift requests d187799490 
All three authentik components were sitting at 70-80% of their 1Gi
memory limit at steady state, with no headroom for spikes
(LDAP sync, blueprint apply, pg checkpoint/vacuum, gunicorn worker
restart). Today's outage during the kubelet config rollout on k8s-2
saw ak-pg-1 (the primary at the time) shut down on a readiness-probe
failure and not come back cleanly; suspected contributing cause is
that brief memory pressure tipped it past the 1Gi limit.

- ak-pg cluster: 512Mi req / 1Gi limit -> 768Mi req / 2Gi limit
  (steady-state usage on primary is ~580MiB; vacuums and checkpoint
  spikes were eating into the cap with no margin)
- authentik-server: 768Mi req / 1Gi limit -> 1Gi req / 2Gi limit
  (live pods sitting at 720-745MiB, ~74% of cap)
- authentik-worker: 768Mi req / 1Gi limit -> 1Gi req / 2Gi limit
  (workers run heavy background tasks - LDAP sync, policy eval,
  blueprint application - and need headroom)

Limit bump is the safety net (no OOM). Request bump matches steady
state so scheduler accounts for real footprint.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.renner merged commit cdb3af9663 into master 2026-06-13 08:21:31 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#14