chore(immich): take server and machine-learning down for the jupiter window #570

2026-08-25T15:36:27Z

claude-bot commented

2026-08-25 15:36:27 +00:00

Temporary — both replica counts go back to 1 once k8s-1 has rejoined. The manifests carry that instruction inline.

With one of three nodes out for the bare-metal conversion, the remaining pair cannot hold the workload. Two independent constraints:

CPU — cluster requests exceed two nodes' capacity by ~200m, once k8s-1's DaemonSets (which die with the node) and mon-c/osd-2 (node-bound, stay unplaced) are correctly discounted.
Disk — k8s-3 has 11.8 GiB free of 59.9 GiB and the kubelet evicts below ~6.0 GiB. It already tripped DiskPressure once today during a three-minute partial drain. Image GC cannot help: talos/patches/cluster.yaml already sets the thresholds to 70/60 and records that dropping high to 60 moved free space by 0.02 percentage points, because ~20 GB is overlayfs snapshots held by running containers.

immich is the right thing to stop rather than squeeze. immich-machine-learning reserves 500m against a measured 2m idle draw — but inference bursts hard, so trimming its request would misrepresent what it actually needs the moment someone uploads photos. Stopping it removes the reservation honestly, and takes its pods and image pulls off k8s-3 for the duration.

Frees 1,000m against a 200m shortfall, so there is real margin rather than a squeeze.

The CNPG Postgres cluster is deliberately left running — no hibernation dance, and recovery is just these two replica counts.

🤖 Generated with Claude Code

**Temporary — both replica counts go back to 1 once k8s-1 has rejoined.** The manifests carry that instruction inline. With one of three nodes out for the bare-metal conversion, the remaining pair cannot hold the workload. Two independent constraints: - **CPU** — cluster requests exceed two nodes' capacity by ~200m, once k8s-1's DaemonSets (which die with the node) and `mon-c`/`osd-2` (node-bound, stay unplaced) are correctly discounted. - **Disk** — k8s-3 has 11.8 GiB free of 59.9 GiB and the kubelet evicts below ~6.0 GiB. It already tripped `DiskPressure` once today during a three-minute partial drain. Image GC cannot help: `talos/patches/cluster.yaml` already sets the thresholds to 70/60 and records that dropping high to 60 moved free space by 0.02 percentage points, because ~20 GB is overlayfs snapshots held by running containers. immich is the right thing to **stop** rather than squeeze. `immich-machine-learning` reserves 500m against a measured 2m idle draw — but inference bursts hard, so trimming its request would misrepresent what it actually needs the moment someone uploads photos. Stopping it removes the reservation honestly, and takes its pods and image pulls off k8s-3 for the duration. Frees 1,000m against a 200m shortfall, so there is real margin rather than a squeeze. The CNPG Postgres cluster is deliberately left running — no hibernation dance, and recovery is just these two replica counts. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-25 15:36:28 +00:00

chore(immich): take server and machine-learning down for the jupiter window

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

6c1f34e992

With one of three nodes out for the bare-metal conversion, the remaining pair
cannot hold the workload: cluster CPU requests exceed two nodes' capacity by
roughly 200m once DaemonSets and unplaceable Ceph pods are discounted, and
k8s-3 sits ~5.8 GiB above the kubelet's eviction threshold with image GC
already at its floor.

immich is the right thing to stop rather than squeeze. machine-learning
reserves 500m against a 2m idle draw, but inference bursts hard, so cutting
its request would misrepresent what it needs the moment someone uploads
photos. Stopping it removes the reservation honestly, and takes its pods and
image pulls off k8s-3 for the duration.

The Postgres cluster stays up, so recovery is these two replica counts and
nothing else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 63fc1ce63e into master

2026-08-25 15:36:49 +00:00

k.renner referenced this issue from a commit

2026-08-25 15:36:50 +00:00

Merge pull request 'chore(immich): take server and machine-learning down for the jupiter window' (#570) from immich-window-scaledown into master

issue-fix-bot referenced this pull request

2026-08-25 16:44:21 +00:00

fix: [alert] CephMonDiskLow (monitoring) (#569) #572

alert-monitor-bot referenced this pull request

2026-08-25 17:34:11 +00:00

[alert] TargetDown (pve-node) #573

alert-monitor-bot referenced this pull request