fix: [alert] KubeDeploymentReplicasMismatch (paperless) (#897) #899

2026-09-21T10:45:07Z

issue-fix-bot commented

2026-09-21 10:45:07 +00:00

What this PR does

KubeDeploymentReplicasMismatch on paperless/paperless is an OOM crash-loop, not a
scheduling or availability problem. ocrmypdf sizes its worker pool from
os.cpu_count(), which reports the host's core count and ignores cgroup limits.
Node k8s-1 has 20 cores and the container had limits.memory: 2Gi with none of
PAPERLESS_TASK_WORKERS / PAPERLESS_THREADS_PER_WORKER set anywhere, so every OCR
run rasterised 20 pages concurrently into a 2 GiB budget and was OOMKilled
(exitCode: 137) ~4 s after OCR began, ~37 s into each container start. Because one
scanned PDF sits in the consume directory and is re-enqueued on every boot, this
loops forever — paperless is fully down, web UI included, not merely degraded. This
PR bounds the OCR worker pool to 4 (from 20) and raises the memory limit to 4Gi as
margin behind that cap.

Changes

kubernetes/apps/paperless/paperless.yaml — in the paperless-config ConfigMap,
add PAPERLESS_THREADS_PER_WORKER: "4" (ocrmypdf's --jobs, previously defaulting
to the node's 20 cores) and PAPERLESS_TASK_WORKERS: "1". Peak memory scales with
the product of the two, so both are pinned — leaving either to an upstream default
leaves the bound undefined. TASK_WORKERS: "1" is the value this version already
resolved to, so it is explicit rather than a behaviour change.
kubernetes/apps/paperless/paperless.yaml — Deployment resources.limits.memory
2Gi → 4Gi, as margin behind the concurrency cap. Deliberately not a fix on
its own: on a 20-core node, raising the limit alone just moves the cliff.

No CPU limit was added. It would not help (os.cpu_count() does not observe CPU
quota, so the worker count would still be 20), and throttling the container's boot —
which can run a checksum backfill and a full index rebuild — risks burning the
10-minute startupProbe budget. That is the "could take it down harder" case, so I
left it alone.

Verification

Verified live this run (all read-only, via ./.ci-diag/kubectl):

-n paperless get pods -o wide → paperless-756f45cd79-tv88m 1/2 CrashLoopBackOff 28 (4m40s ago). Still looping, and restarts have grown from the 14 recorded in the
issue to 28 — the condition does not self-heal.
get pod ... -o jsonpath='{...containerStatuses...}' → paperless:
"exitCode":137,"reason":"OOMKilled","startedAt":"…10:37:07Z","finishedAt":"…10:37:44Z"
— a 37-second life. Sidecar bridge-tls-proxy has restarts=0, so only the
paperless container is affected (hence 1/2).
get node k8s-1 -o jsonpath='{.status.capacity.cpu}' → 20.
logs … -c paperless --previous --tail=120 — the decisive line, and it ties the two
together:
```
12:37:40,838 [ocrmypdf._pipelines.ocr] [77cd6e2d] Starting processing with 20 workers concurrently
```
followed by page-rotation lines up to 12:37:44,258, where the log simply stops.
12:37:44 Europe/Berlin == the container's finishedAt: 10:37:44Z, to the second.
The same log also prints concurrency: 1 (prefork) in the celery banner, which is
what pins TASK_WORKERS to 1 and makes THREADS_PER_WORKER default to
floor(20/1) = 20 — the exact number ocrmypdf reports.
get cm paperless-config -o jsonpath=… for all three settings → empty, confirming
they are unset in the live cluster, matching a repo-wide grep that found no matches.
describe node k8s-1 → MemoryPressure False, 98 GiB capacity, memory requests at
13759Mi (14%). The extra 2Gi of limit is comfortably absorbed.

After merge: Flux reconciles the ConfigMap; configmap.reloader.stakater.com/reload: paperless-config on the Deployment rolls the pod automatically (this annotation is
already present — a ConfigMap-only change would otherwise not be picked up by
envFrom). Confirm with kubectl -n paperless logs deploy/paperless -c paperless | grep "workers concurrently" → should read 4 workers, not 20.
KubeDeploymentReplicasMismatch should clear once the pod reaches Ready.

Notes

The repo change alone will not stop the loop — a human must also clear the consume
directory. The triggering file is at
/usr/src/paperless/consume/scanner/20260921_101949_ADS-1700W_002274.pdf on the
paperless-consume PVC. Recommended: move it aside, let the pod come up healthy, then
re-feed it and watch memory.

Correction to the issue's mechanism. The issue states that "Celery redelivers the
identical task" and that remediation requires "clearing the redelivered Celery task
from Redis". The startup log shows that is not what happens:

[paperless.management.consumer] Processing existing files in /usr/src/paperless/consume
[paperless.management.consumer] Adding /usr/src/paperless/consume/scanner/20260921_101949_ADS-1700W_002274.pdf to the task queue

Each attempt is a fresh enqueue from paperless's startup directory scan, which is
why every attempt carries a different task id (the issue noticed the differing ids but
read them as redeliveries). The consequence is practical: no Redis surgery is needed,
and removing the file from the consume directory is both necessary and sufficient to
break the loop. The issue's conclusion — that a human must act on the cluster — stands;
only the required step is simpler than described.

Not verified. Why this document and not the previous one (page count, DPI, image
size) is still unestablished, as the issue noted — I did not settle it, since the fix
does not depend on it. Consequently I cannot prove 4 workers × 4Gi is enough for this
specific file; it is a ~10× aggregate improvement (5× fewer concurrent rasterisations,
2× the limit), which makes it very likely but not certain. If it still OOMs after this
lands, the next lever is PAPERLESS_THREADS_PER_WORKER: "2" and/or
PAPERLESS_OCR_MAX_IMAGE_PIXELS, not a further memory bump.

I did not re-derive the issue's note about container_memory_working_set_bytes peaking
at ~825 MB; its reasoning is sound — a 30 s cadvisor scrape cannot see a spike in a
container that lives 37 s and blows up in its last 4 s — and the cgroup's
reason: OOMKilled is authoritative regardless.

The issue's unrelated tail (FailedScheduling / VolumeFailedDelete in woodpecker-ci,
velero, default) is out of scope here and untouched. Nothing in the issue asked for
changes outside the edit scope.

Closes #897

## What this PR does `KubeDeploymentReplicasMismatch` on `paperless/paperless` is an OOM crash-loop, not a scheduling or availability problem. ocrmypdf sizes its worker pool from `os.cpu_count()`, which reports the **host's** core count and ignores cgroup limits. Node `k8s-1` has 20 cores and the container had `limits.memory: 2Gi` with none of `PAPERLESS_TASK_WORKERS` / `PAPERLESS_THREADS_PER_WORKER` set anywhere, so every OCR run rasterised 20 pages concurrently into a 2 GiB budget and was `OOMKilled` (`exitCode: 137`) ~4 s after OCR began, ~37 s into each container start. Because one scanned PDF sits in the consume directory and is re-enqueued on every boot, this loops forever — paperless is fully down, web UI included, not merely degraded. This PR bounds the OCR worker pool to 4 (from 20) and raises the memory limit to 4Gi as margin behind that cap. ## Changes - `kubernetes/apps/paperless/paperless.yaml` — in the `paperless-config` ConfigMap, add `PAPERLESS_THREADS_PER_WORKER: "4"` (ocrmypdf's `--jobs`, previously defaulting to the node's 20 cores) and `PAPERLESS_TASK_WORKERS: "1"`. Peak memory scales with the *product* of the two, so both are pinned — leaving either to an upstream default leaves the bound undefined. `TASK_WORKERS: "1"` is the value this version already resolved to, so it is explicit rather than a behaviour change. - `kubernetes/apps/paperless/paperless.yaml` — Deployment `resources.limits.memory` `2Gi` → `4Gi`, as margin behind the concurrency cap. Deliberately **not** a fix on its own: on a 20-core node, raising the limit alone just moves the cliff. No CPU limit was added. It would not help (`os.cpu_count()` does not observe CPU quota, so the worker count would still be 20), and throttling the container's boot — which can run a checksum backfill and a full index rebuild — risks burning the 10-minute `startupProbe` budget. That is the "could take it down harder" case, so I left it alone. ## Verification Verified live this run (all read-only, via `./.ci-diag/kubectl`): - `-n paperless get pods -o wide` → `paperless-756f45cd79-tv88m 1/2 CrashLoopBackOff 28 (4m40s ago)`. Still looping, and restarts have grown from the 14 recorded in the issue to 28 — the condition does not self-heal. - `get pod ... -o jsonpath='{...containerStatuses...}'` → `paperless`: `"exitCode":137,"reason":"OOMKilled","startedAt":"…10:37:07Z","finishedAt":"…10:37:44Z"` — a 37-second life. Sidecar `bridge-tls-proxy` has `restarts=0`, so only the `paperless` container is affected (hence `1/2`). - `get node k8s-1 -o jsonpath='{.status.capacity.cpu}'` → `20`. - `logs … -c paperless --previous --tail=120` — the decisive line, and it ties the two together: ``` 12:37:40,838 [ocrmypdf._pipelines.ocr] [77cd6e2d] Starting processing with 20 workers concurrently ``` followed by page-rotation lines up to `12:37:44,258`, where the log simply stops. `12:37:44` Europe/Berlin == the container's `finishedAt: 10:37:44Z`, to the second. The same log also prints `concurrency: 1 (prefork)` in the celery banner, which is what pins `TASK_WORKERS` to 1 and makes `THREADS_PER_WORKER` default to `floor(20/1) = 20` — the exact number ocrmypdf reports. - `get cm paperless-config -o jsonpath=…` for all three settings → empty, confirming they are unset in the live cluster, matching a repo-wide grep that found no matches. - `describe node k8s-1` → `MemoryPressure False`, 98 GiB capacity, memory requests at 13759Mi (14%). The extra 2Gi of limit is comfortably absorbed. After merge: Flux reconciles the ConfigMap; `configmap.reloader.stakater.com/reload: paperless-config` on the Deployment rolls the pod automatically (this annotation is already present — a ConfigMap-only change would otherwise not be picked up by `envFrom`). Confirm with `kubectl -n paperless logs deploy/paperless -c paperless | grep "workers concurrently"` → should read `4 workers`, not `20`. `KubeDeploymentReplicasMismatch` should clear once the pod reaches Ready. ## Notes **The repo change alone will not stop the loop — a human must also clear the consume directory.** The triggering file is at `/usr/src/paperless/consume/scanner/20260921_101949_ADS-1700W_002274.pdf` on the `paperless-consume` PVC. Recommended: move it aside, let the pod come up healthy, then re-feed it and watch memory. **Correction to the issue's mechanism.** The issue states that "Celery redelivers the identical task" and that remediation requires "clearing the redelivered Celery task from Redis". The startup log shows that is not what happens: ``` [paperless.management.consumer] Processing existing files in /usr/src/paperless/consume [paperless.management.consumer] Adding /usr/src/paperless/consume/scanner/20260921_101949_ADS-1700W_002274.pdf to the task queue ``` Each attempt is a **fresh enqueue from paperless's startup directory scan**, which is why every attempt carries a different task id (the issue noticed the differing ids but read them as redeliveries). The consequence is practical: no Redis surgery is needed, and removing the file from the consume directory is both necessary and sufficient to break the loop. The issue's conclusion — that a human must act on the cluster — stands; only the required step is simpler than described. **Not verified.** Why *this* document and not the previous one (page count, DPI, image size) is still unestablished, as the issue noted — I did not settle it, since the fix does not depend on it. Consequently I cannot *prove* 4 workers × 4Gi is enough for this specific file; it is a ~10× aggregate improvement (5× fewer concurrent rasterisations, 2× the limit), which makes it very likely but not certain. If it still OOMs after this lands, the next lever is `PAPERLESS_THREADS_PER_WORKER: "2"` and/or `PAPERLESS_OCR_MAX_IMAGE_PIXELS`, not a further memory bump. I did not re-derive the issue's note about `container_memory_working_set_bytes` peaking at ~825 MB; its reasoning is sound — a 30 s cadvisor scrape cannot see a spike in a container that lives 37 s and blows up in its last 4 s — and the cgroup's `reason: OOMKilled` is authoritative regardless. The issue's unrelated tail (FailedScheduling / VolumeFailedDelete in `woodpecker-ci`, `velero`, `default`) is out of scope here and untouched. Nothing in the issue asked for changes outside the edit scope. Closes #897

issue-fix-bot added 1 commit 2026-09-21 10:45:07 +00:00

fix: [alert] KubeDeploymentReplicasMismatch (paperless) (#897 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

4c2e1e5750

issue-fix-bot referenced this pull request

2026-09-21 10:45:07 +00:00

[alert] KubeDeploymentReplicasMismatch (paperless) #897

review-bot reviewed

2026-09-21 10:47:11 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 4c2e1e575094ab09ac0cbfa295e2abfb6ab2b81b.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes an OOM crash-loop in paperless OCR by bounding the worker pool concurrency to 4 (from the uncontrolled default of 20 based on host CPU count) and increasing the memory limit from 2Gi to 4Gi. The root cause analysis is thorough and well-documented, with detailed verification steps provided.

🔴 Blocking

None. The fix is sound and correctly targeted.

🟡 Suggestions

ConfigMap Reloader annotation location (lines 116-126): The configmap.reloader.stakater.com/reload: paperless-config annotation is placed at the Deployment metadata level, not at the pod template metadata level (lines 134-137). Stakater Reloader typically watches annotations in spec.template.metadata.annotations to trigger pod restarts. While the PR body states this annotation is "already present," verify that Reloader actually triggers a pod restart when the ConfigMap changes. If it doesn't work as currently configured, the annotation should be moved to line 137 (after the labels) in the pod template metadata section.

🟢 Nits

Comment clarity (lines 30-46): The comment is thorough and accurate, but it's quite long (17 lines). This is acceptable for a critical constraint, but it might be worth confirming the conciseness is intentional rather than over-documented.

Tests

This is a Kubernetes configuration file, not application code. No unit tests are applicable. The PR body includes detailed manual verification steps confirming:

Current state: 20 concurrent workers causing OOMKilled after ~37s
Node capacity: 20 cores on k8s-1, 98 GiB memory available
Post-merge verification plan: Check logs for "4 workers" instead of "20"

The fix is appropriate for infrastructure-as-code and requires operational verification post-merge (clearing the consume directory and monitoring).

Recommendation

Looks reasonable — no blocking issues; comments above are optional. Verify post-merge that the ConfigMap reloader annotation triggers a pod restart, and monitor the first container run to confirm the new memory/concurrency settings prevent the OOM crash.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `4c2e1e575094ab09ac0cbfa295e2abfb6ab2b81b`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes an OOM crash-loop in paperless OCR by bounding the worker pool concurrency to 4 (from the uncontrolled default of 20 based on host CPU count) and increasing the memory limit from 2Gi to 4Gi. The root cause analysis is thorough and well-documented, with detailed verification steps provided. ## 🔴 Blocking None. The fix is sound and correctly targeted. ## 🟡 Suggestions 1. **ConfigMap Reloader annotation location** (lines 116-126): The `configmap.reloader.stakater.com/reload: paperless-config` annotation is placed at the Deployment metadata level, not at the pod template metadata level (lines 134-137). Stakater Reloader typically watches annotations in `spec.template.metadata.annotations` to trigger pod restarts. While the PR body states this annotation is "already present," verify that Reloader actually triggers a pod restart when the ConfigMap changes. If it doesn't work as currently configured, the annotation should be moved to line 137 (after the labels) in the pod template metadata section. ## 🟢 Nits 1. **Comment clarity** (lines 30-46): The comment is thorough and accurate, but it's quite long (17 lines). This is acceptable for a critical constraint, but it might be worth confirming the conciseness is intentional rather than over-documented. ## Tests This is a Kubernetes configuration file, not application code. No unit tests are applicable. The PR body includes detailed manual verification steps confirming: - Current state: 20 concurrent workers causing OOMKilled after ~37s - Node capacity: 20 cores on k8s-1, 98 GiB memory available - Post-merge verification plan: Check logs for "4 workers" instead of "20" The fix is appropriate for infrastructure-as-code and requires operational verification post-merge (clearing the consume directory and monitoring). ## Recommendation Looks reasonable — no blocking issues; comments above are optional. Verify post-merge that the ConfigMap reloader annotation triggers a pod restart, and monitor the first container run to confirm the new memory/concurrency settings prevent the OOM crash.

k.renner closed this pull request

2026-09-23 04:54:54 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#899