fix(paperless-eval): bound generation server-side; the model does not always stop

Three of the first eleven documents in a run never terminated. One reached
10269 tokens on a classification that needs ~300, at a perfectly healthy
11 tok/s — so it read as slowness and was neither slow nor a context problem.

llama-server's --n-predict defaults to -1, infinity, and paperless sets no
token budget of its own and waits. Nothing anywhere bounded the request. Four
documents cost most of an hour.

--n-predict 768 on the sidecar caps it, and a truncated response is better
evidence than a timeout: it shows what the model was doing instead of
stopping. The client timeout drops to 300 s and becomes a backstop.

Also:

- the degeneracy check fired on 'Rechnung 2025-5425541' — 8 letters in 21
  characters, below the 40% Latin threshold. Digits and punctuation are normal
  in a title; look for characters from other scripts directly instead of
  inferring them from a shortage of Latin ones.

- the ctx-size comment still carried the claim bisection disproved in #777,
  that 32768 corrupts inference. The boundary is prompt length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/llama-server/deployment.yaml

@@ -75,19 +75,37 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 16k, and going above it corrupts inference rather than failing.
-            # At 32768 per slot this server answered a 650-token classification
-            # prompt with 1025 tokens of "!" — the KV cache producing numbers
-            # rather than language. It does not crash, it does not warn, and
-            # the OpenAI response looks well-formed, so the damage only shows
-            # in what callers get back: Paperless ran past its 300 s timeout,
-            # and Home Assistant got an empty completion and a hallucinated
-            # device list.
+            # 16k is room to spare, not a safety limit. The limit that matters
+            # is not set here at all.
             #
-            # The same Vulkan/Mesa stack already fails hard at --ubatch-size
-            # 2048; this is the quiet version of that limit. Raise this only
-            # against a generation that is read, not against a server that
-            # merely starts.
+            # Isolated by bisection on a scratch pod: this stack breaks on
+            # PROMPT length, and the value of --ctx-size makes no difference to
+            # it. Measured at --ctx-size 65536 with a single slot, so with four
+            # times the headroom:
+            #
+            #     13588 prompt tokens   coherent answer
+            #     15086                 one token, immediate EOS, empty content
+            #     16088                 CJK characters and digit runs
+            #
+            # A single request past that boundary then poisons the slot: the
+            # same 13588-token prompt that answered cleanly a minute earlier
+            # returns one empty token afterwards, and stays broken until the
+            # pod restarts. That is why it looked intermittent, and why a
+            # 650-token classification once came back as 1025 tokens of "!" —
+            # it followed Home Assistant's ~19k prompts through the same slot.
+            #
+            # Nothing reports any of it: finish_reason "stop", a well-formed
+            # OpenAI response, HTTP 200, /health green.
+            #
+            # So the operating rule is about callers, not this flag. Paperless
+            # sits near 5k and is safe; Home Assistant's Assist prompt measured
+            # 18910 and never had a chance. An earlier version of this comment
+            # blamed --ctx-size 32768, which the bisection disproves — 32768
+            # and 65536 both answer correctly as long as the prompt stays under
+            # the boundary.
+            #
+            # The same stack also fails hard at --ubatch-size 2048, which is
+            # the loud version of a limit this one keeps quiet about.
             #
             - --ctx-size
             - "16384"

kubernetes/apps/paperless-eval/cluster.yaml

@@ -0,0 +1,116 @@
+---
+# A read-only view of paperless's backup store, used as a recovery source and
+# nothing else.
+#
+# It points at the production prefix, which is the whole point — the eval
+# instance is a copy of the real archive's metadata — and that makes what is
+# ABSENT here the safety-critical part:
+#
+#   no retentionPolicy   The production store has "30d". Without one here
+#                        there is no policy to enforce, and enforcement is not
+#                        tied to taking a backup: the plugin runs it from the
+#                        instance sidecar on a timer
+#                        (instanceSidecarConfiguration.retentionPolicyIntervalSeconds,
+#                        default 1800). So "this cluster never backs up" would
+#                        not have been enough on its own.
+#
+# A second property makes it belt-and-braces. Retention runs
+# barman-cloud-backup-delete, which takes a server name and works inside
+# <destinationPath>/<serverName> — the store's own status keys its recovery
+# window by "pprl-pg", confirming that layout. This cluster is pprleval-pg, so
+# even an accidental retention policy would address a prefix that holds nothing
+# rather than production's backups.
+#
+# The cluster below completes the guarantee with isWALArchiver: false, so
+# nothing is written back either. It does load the plugin — that is required
+# for the restore to work at all — and read-only comes from the flag rather
+# than from leaving the plugin out.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: pprleval-pg-source
+  namespace: paperless-eval
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+---
+# A disposable clone of the paperless database, for measuring how well the
+# local LLM classifies documents without running the experiment against the
+# live archive.
+#
+# Losing this cluster costs a re-restore and nothing else, which is the intent
+# — it is rebuilt whenever a test needs a clean starting point.
+#
+# One instance, not two. Replication buys availability, and an eval instance
+# that is down is simply an eval that waits.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: pprleval-pg
+  namespace: paperless-eval
+spec:
+  instances: 1
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  enableSuperuserAccess: true
+  storage:
+    # Delete-reclaim, unlike the Retain default: this volume is a copy that
+    # gets rebuilt, so it should vanish with the cluster rather than leave a
+    # Released PV and an orphaned RBD image behind (issue #262).
+    storageClass: ceph-block-ephemeral
+    size: 10Gi
+  # Declared with isWALArchiver: false, which is what kubernetes/RECOVERY.md
+  # arrived at after an earlier version of that runbook said to omit it: the
+  # instance manager loads this plugin to replay WAL during the restore, and
+  # without it the instance never starts ("Unknown plugin:
+  # barman-cloud.cloudnative-pg.io").
+  #
+  # externalClusters below only says WHERE the backup is; it does not get the
+  # plugin injected into the pod. So "no plugin" is not the way to keep this
+  # read-only — "plugin, not archiving" is, and it is the combination verified
+  # on im-pg on 2026-07-25.
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      enabled: true
+      isWALArchiver: false
+      parameters:
+        barmanObjectName: pprleval-pg-source
+  bootstrap:
+    recovery:
+      source: pprl-pg
+      database: paperless
+      owner: paperless
+  externalClusters:
+    - name: pprl-pg
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: pprleval-pg-source
+          serverName: pprl-pg
+  postgresql:
+    parameters:
+      shared_buffers: "256MB"
+      max_connections: "100"
+      # The production tuning for write reduction is deliberately not copied.
+      # It exists to spare the SSDs under a database that runs for years; this
+      # one is read-only for hours at a time and then deleted.
+  resources:
+    requests:
+      cpu: 200m
+      memory: 512Mi
+    limits:
+      memory: 1Gi

kubernetes/apps/paperless-eval/kustomization.yaml

@@ -0,0 +1,13 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: paperless-eval
+# The instance only. The measurement harness lives in scripts/paperless-eval/
+# and is applied by hand — a Job is immutable, re-running one means deleting and
+# recreating it, and Flux would just put the old one back. Nothing here should
+# converge on a finished measurement.
+resources:
+  - secrets.yaml
+  - cluster.yaml
+  - pvcs.yaml
+  - redis.yaml
+  - paperless.yaml

kubernetes/apps/paperless-eval/paperless.yaml

@@ -0,0 +1,203 @@
+---
+# Deliberately much smaller than the production ConfigMap. Everything that
+# exists to serve people is gone: no remote-user auth, no CSRF origins, no
+# barcode or mail consumption, no Tika/Gotenberg. Nothing consumes documents
+# here, and nothing signs in.
+#
+# What remains is the database connection and the AI settings, because those
+# are what the evaluation measures.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: paperless-eval-config
+  namespace: paperless-eval
+data:
+  PAPERLESS_URL: "http://paperless.paperless-eval.svc.cluster.local"
+  PAPERLESS_ALLOWED_HOSTS: "*"
+  PAPERLESS_TIME_ZONE: "Europe/Berlin"
+  PAPERLESS_REDIS: "redis://redis:6379"
+  PAPERLESS_DBENGINE: "postgresql"
+  PAPERLESS_DBHOST: "pprleval-pg-rw"
+  PAPERLESS_DBNAME: "paperless"
+  # Polling off, so a pod that finds an empty consume volume does not spend a
+  # thread looking at it every ten seconds for the length of a run.
+  PAPERLESS_CONSUMER_POLLING_INTERVAL: "0"
+
+  # The AI settings under test. The endpoints are placeholders that each eval
+  # job overrides for the model it is measuring — a job runs its own
+  # llama-server beside paperless in the same pod, so the value that matters at
+  # run time is the one in the job, not the one here.
+  #
+  # Same caveat as production, and sharper here: paperless reads these from an
+  # ApplicationConfiguration row and falls back to the environment only when
+  # that row is empty. The row comes across in the restored database, so
+  # anything configured in the production admin UI arrives with it and silently
+  # wins over this ConfigMap — an eval would then measure production's settings
+  # while appearing to measure the job's.
+  #
+  # Check it before trusting any result, and after every fresh restore:
+  #
+  #   select llm_backend, llm_model, llm_embedding_backend, llm_embedding_model
+  #     from paperless_applicationconfiguration;
+  #
+  # Every column must be null for the environment to apply.
+  PAPERLESS_AI_ENABLED: "true"
+  PAPERLESS_AI_LLM_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_ENDPOINT: "http://127.0.0.1:8080/v1"
+  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
+  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
+  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
+  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "900"
+  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
+  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
+  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
+---
+# A base instance that proves the restored database is usable and that the
+# search index and classifier build against it. The evaluation itself does not
+# run here — each job brings its own paperless container so it can point at its
+# own model — but this is what makes a broken clone visible before a job spends
+# hours on it.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless
+  namespace: paperless-eval
+  labels:
+    app.kubernetes.io/name: paperless
+  annotations:
+    configmap.reloader.stakater.com/reload: paperless-eval-config
+    # Both, like production: the secret refreshes on the same 10m cycle, and
+    # without this the pod would keep a rotated key until something else
+    # restarted it — which defeats the point of matching the interval.
+    secret.reloader.stakater.com/reload: paperless-eval-secrets
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperless
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: paperless
+          # No $imagepolicy marker, unlike apps/paperless: a measurement is
+          # only comparable against a fixed version, so this tag moves when
+          # somebody decides to re-measure, not when a release lands.
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
+          ports:
+            - name: http
+              containerPort: 8000
+          envFrom:
+            - configMapRef:
+                name: paperless-eval-config
+          env:
+            - name: PAPERLESS_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-eval-secrets
+                  key: secret-key
+            - name: PAPERLESS_DBUSER
+              valueFrom:
+                secretKeyRef:
+                  name: pprleval-pg-app
+                  key: username
+            - name: PAPERLESS_DBPASS
+              valueFrom:
+                secretKeyRef:
+                  name: pprleval-pg-app
+                  key: password
+          volumeMounts:
+            - name: data
+              mountPath: /usr/src/paperless/data
+            - name: media
+              mountPath: /usr/src/paperless/media
+            - name: consume
+              mountPath: /usr/src/paperless/consume
+          # 60 minutes, against production's 10, and matching this app's Flux
+          # Kustomization timeout.
+          #
+          # Production's budget was sized for a from-scratch boot on this same
+          # corpus and held, so 10 would probably do — but there the volume is
+          # Retain and the index survives, making that a one-off. Here the
+          # volume is ephemeral and the index is rebuilt on every re-test, so
+          # the expensive path is the normal one. A killed pod starts the work
+          # over rather than resuming, which turns a slow boot into a
+          # crash-loop that stalls Flux behind wait: true.
+          #
+          # The LLM index is not in this budget: it is a celery task, and boot
+          # only runs init-llmindex-migrate, a migration check.
+          startupProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            periodSeconds: 20
+            timeoutSeconds: 5
+            failureThreshold: 180
+          livenessProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            initialDelaySeconds: 120
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              memory: 2Gi
+          # Production's context verbatim, and it has to be: the paperless-ngx
+          # entrypoint starts as root, chowns the mounted data/media/consume
+          # volumes and then drops to uid 1000 via setgid/setuid. Without these
+          # the same image fails that sequence with EPERM.
+          #
+          # It bites harder here than in production, where those volumes were
+          # chowned once years ago and keep their ownership. These are fresh on
+          # every rebuild, so the chown is not a no-op.
+          securityContext:
+            runAsNonRoot: false
+            allowPrivilegeEscalation: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: paperless-eval-golden
+        - name: media
+          persistentVolumeClaim:
+            claimName: paperless-eval-media
+        - name: consume
+          persistentVolumeClaim:
+            claimName: paperless-eval-consume
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless
+  namespace: paperless-eval
+spec:
+  selector:
+    app.kubernetes.io/name: paperless
+  ports:
+    - port: 80
+      targetPort: http

kubernetes/apps/paperless-eval/pvcs.yaml

@@ -0,0 +1,129 @@
+---
+# All four use ceph-block-ephemeral rather than the Retain default. Everything
+# here is derived — a restored database, an index built from it, a cache —  so
+# the volumes should go when the namespace does instead of leaving Released PVs
+# and orphaned RBD images to reclaim by hand (issue #262).
+#
+# /data holds the Tantivy search index and the classifier pickle, both rebuilt
+# from the database, and the LLM index — which is the expensive one at roughly
+# three hours for the full corpus.
+#
+# Measured on production for the same ~3900 documents: 36 MiB search index,
+# 125 MiB LLM index. 10Gi is far more than that needs, and deliberately so —
+# an embedding model with wider vectors than bge-m3's 1024 dimensions is
+# exactly what phase two varies, and running out of space mid-run would cost
+# the whole rebuild.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  # Named for its role rather than its mount point: the smoke-test deployment
+  # builds /data here, and every eval job copies it out. Replaces the earlier
+  # RWO paperless-eval-data, whose access mode could not be widened in place —
+  # a bound claim's spec is immutable.
+  name: paperless-eval-golden
+  namespace: paperless-eval
+  labels:
+    velero.io/exclude-from-backup: "true"
+spec:
+  # RWX, because the eval jobs read it too — each copies it into its own
+  # emptyDir rather than mounting it live. The vector store is SQLite, and
+  # SQLite locking across pods over CephFS is not something to rely on.
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem-ephemeral
+  resources:
+    requests:
+      storage: 10Gi
+---
+# Empty, and stays that way. Paperless refuses to start without the path, but
+# the AI path reads title, content and filename out of the database and never
+# opens a file — so the documents themselves are not copied here. That is what
+# keeps this instance cheap and keeps a second copy of the archive off the
+# cluster.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-eval-media
+  namespace: paperless-eval
+  labels:
+    velero.io/exclude-from-backup: "true"
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block-ephemeral
+  resources:
+    requests:
+      storage: 1Gi
+---
+# Also empty: nothing is consumed here, but the path has to exist.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-eval-consume
+  namespace: paperless-eval
+  labels:
+    velero.io/exclude-from-backup: "true"
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block-ephemeral
+  resources:
+    requests:
+      storage: 1Gi
+---
+# The eval jobs' model cache, RWX so several jobs can read the same weights at
+# once — that is the whole point of measuring two candidates in parallel.
+#
+# Separate from the ai namespace's llama-server-models despite holding some of
+# the same files: PVCs do not cross namespaces. A model therefore downloads
+# once here and is then shared by every job, which is the cost a new candidate
+# carries anyway.
+#
+# NOT ephemeral, unlike everything else in this namespace: re-downloading tens
+# of gigabytes on a slow uplink is a worse outcome than an orphaned volume.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: llama-models
+  namespace: paperless-eval
+  labels:
+    velero.io/exclude-from-backup: "true"
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem
+  resources:
+    requests:
+      storage: 120Gi
+---
+# Results outlive the jobs that produce them: a Job is deleted and recreated to
+# re-run a configuration, and comparing configurations is the entire exercise.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-eval-results
+  namespace: paperless-eval
+  labels:
+    velero.io/exclude-from-backup: "true"
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-eval-redis
+  namespace: paperless-eval
+  labels:
+    velero.io/exclude-from-backup: "true"
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block-ephemeral
+  resources:
+    requests:
+      storage: 1Gi

kubernetes/apps/paperless-eval/redis.yaml

@@ -0,0 +1,81 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  namespace: paperless-eval
+  labels:
+    app.kubernetes.io/name: redis
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: redis
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: redis
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 999
+        fsGroup: 999
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: redis
+          # No $imagepolicy marker; see the note on the paperless image.
+          image: redis:8.10.1-alpine
+          # No --save: this holds the celery queue and the suggestion cache,
+          # both of which are regenerated. Persisting them across a restart of
+          # a throwaway instance would only preserve stale suggestions.
+          args:
+            - redis-server
+            - --save
+            - ""
+            - --loglevel
+            - warning
+          ports:
+            - name: redis
+              containerPort: 6379
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          livenessProbe:
+            tcpSocket:
+              port: redis
+            initialDelaySeconds: 10
+          readinessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: paperless-eval-redis
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: paperless-eval
+spec:
+  selector:
+    app.kubernetes.io/name: redis
+  ports:
+    - port: 6379
+      targetPort: redis

kubernetes/apps/paperless-eval/secrets.yaml

@@ -0,0 +1,45 @@
+---
+# The same Hetzner S3 key every CNPG-using namespace pulls, needed here only so
+# the recovery source can be read. See apps/paperless for why one shared key
+# rather than per-app paths.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: paperless-eval
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: infrastructure/hetzner-s3
+---
+# Django's secret key, shared with the production instance rather than given
+# its own vault entry.
+#
+# It signs sessions and CSRF tokens, and this instance has no ingress and no
+# users — nothing signs in, so nothing is at stake in sharing it. A separate
+# entry would mean a vault write before the namespace can start, for a
+# throwaway instance that gets rebuilt.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: paperless-eval-secrets
+  namespace: paperless-eval
+spec:
+  # 10m, matching apps/paperless rather than the 1h default: the point of
+  # sharing the key is that the two hold the same value, and a slower refresh
+  # here would let them diverge for up to an hour after a rotation.
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: paperless-eval-secrets
+  dataFrom:
+    - extract:
+        key: apps/paperless

kubernetes/clusters/homelab/paperless-eval.yaml

@@ -0,0 +1,34 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: paperless-eval
+  namespace: flux-system
+spec:
+  # A throwaway copy of paperless for measuring the local LLM against the real
+  # corpus. Its Cluster restores from paperless's backup store and archives
+  # nothing back; see kubernetes/apps/paperless-eval/cluster.yaml.
+  #
+  # prune: true and no prune-disabled annotations anywhere underneath — the
+  # data here is a copy, and being able to delete the whole namespace by
+  # removing this file is the point.
+  interval: 10m
+  retryInterval: 1m
+  # 30m is not enough: the first start restores the database and then builds
+  # the search index over ~3900 documents before the webserver answers.
+  timeout: 60m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/paperless-eval
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster
+    - name: infrastructure-cnpg-barman
+  healthCheckExprs:
+    - apiVersion: postgresql.cnpg.io/v1
+      kind: Cluster
+      failed: "has(status.phase) && status.phase.startsWith('Failed')"
+      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -69,6 +69,13 @@ kind: Namespace
 metadata:
   name: paperless
 ---
+# A disposable copy of paperless's database, for measuring how the local LLM
+# classifies documents without experimenting on the live archive.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: paperless-eval
+---
 apiVersion: v1
 kind: Namespace
 metadata:

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -34,6 +34,22 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
+    # The paperless-eval namespace reaches the embedding endpoint for the same
+    # reason paperless does; its chat model runs inside each eval job's own pod
+    # and needs no grant here.
+    #
+    # Scoped to the namespace rather than to an app label, unlike the grants
+    # around it: that namespace exists only to run evaluations, and its
+    # workload is jobs created per measurement whose labels vary with what is
+    # being measured. A label match would be a trap that presents as an
+    # embedding call silently dropped.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperless-eval
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     # Home Assistant's llama.cpp conversation agent calls the chat model only —
     # it has no use for the embedding endpoint, so unlike paperless above this
     # grant does not extend to llama-embed.

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -6,6 +6,7 @@ resources:
   - ingress-isolation.yaml
   - authentik-ns.yaml
   - paperless-ns.yaml
+  - paperless-eval-ns.yaml
   - audiobookshelf-ns.yaml
   - home-assistant-ns.yaml
   - spegel-ns.yaml

kubernetes/infrastructure/network-policies/paperless-eval-ns.yaml

@@ -0,0 +1,141 @@
+# Narrower than paperless-ns.yaml, because this namespace serves nobody.
+#
+# There is no ingress, no gate outpost and no hostname: the only things that
+# talk to it are its own pods. Monitoring is admitted so a run can be watched,
+# and host/remote-node for the kubelet probes.
+#
+# Egress reaches the ai namespace for the embedding endpoint, which is the one
+# dependency an eval job has outside its own pod, and world:443 so the CNPG
+# recovery can read the backup store on Hetzner.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: paperless-eval
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperless-eval
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
+    # home-assistant-ns.yaml for the rationale.
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperless-eval
+    # The embedding endpoint. The chat model runs inside each job's own pod, so
+    # it needs no grant; the embedder is shared with production because it is
+    # read-only and cheap, and swapping it is a phase-two concern.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-embed
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": rook-ceph
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+    - toEntities:
+        - host
+        - remote-node
+    # world:443 — the CNPG recovery reads the backup store on Hetzner Object
+    # Storage, and image pulls.
+    - toEntities:
+        - world
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP
+    # ICMP — egress mirror; see ingress block above.
+    - toEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+---
+# The CNPG operator polls each instance's status endpoint over HTTPS
+# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
+# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
+# while Postgres serves normally — issue #433, where the operator's dial timed
+# out because the apps baseline never allowed the database namespace.
+#
+# It matters here even for a throwaway: the Flux Kustomization waits on
+# Cluster status, and a smoke test that cannot report its own health is not one.
+#
+# Scoped to the pprleval-pg endpoints and that one port, like the equivalent
+# rules for wp-pg and ha-pg: the barman and mariadb operators sharing the
+# database namespace have no reason to reach anything here.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pprleval-pg-allow-cnpg-operator
+  namespace: paperless-eval
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: pprleval-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+            app.kubernetes.io/name: cloudnative-pg
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -27,6 +27,7 @@ spec:
         - ingress
         - monitoring
         - paperless
+        - paperless-eval # same apps/paperless key plus the shared barman S3 creds
         - trmnl
         - velero
         - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)

scripts/paperless-eval/README.md

@@ -0,0 +1,72 @@
+# Measuring Paperless suggestions against the real corpus
+
+Compares the two engines Paperless ships — the scikit-learn classifier and the
+LLM path — over documents whose labels a human already chose, in the
+`paperless-eval` namespace rather than against the live archive.
+
+**Applied by hand, not by Flux.** A Job is immutable, so re-running a
+configuration means deleting and recreating it, and a Flux-managed Job would
+simply come back. The namespace itself *is* managed
+(`kubernetes/apps/paperless-eval/`); only the measurement is not.
+
+## Running one
+
+```sh
+kubectl -n paperless-eval create configmap paperless-eval-script \
+  --from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
+  | kubectl apply -f -
+
+kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
+kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
+```
+
+Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
+document) and `<label>.summary.json`, and in the pod log.
+
+Re-running after editing `eval.py` means re-creating the ConfigMap and deleting
+the Job first — `kubectl -n paperless-eval delete job eval-qwen35`.
+
+## Adding a candidate
+
+Copy `job-qwen35.yaml`, then change four things: the Job name, the
+`--hf-repo`/`--alias` on the `llama-server` sidecar, `PAPERLESS_AI_LLM_MODEL` on
+the `paperless` sidecar (it must match the alias), and `EVAL_LABEL` on the
+`eval` container (it names the output files).
+
+Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1
+and k8s-2 have a free iGPU. A third waits.
+
+## Reading the output
+
+`complement` in the summary is the number the exercise exists for. The two
+engines are not alternatives: the classifier costs 0.6 s per document and keeps
+running whatever the LLM does, so what matters is what the LLM adds where the
+classifier is wrong — `llm_rescued` against `llm_only_wrong`. Two aggregate
+percentages cannot show that; 74% against 72% could overlap almost entirely or
+barely at all.
+
+`correspondent_offered` against `correspondent_recognised` is the
+`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
+rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
+stored "Generali AG".
+
+`type_specific` against `type_catchall` splits on whether the true type is
+"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
+semantic content. The classifier can memorise it; a model reading the document
+has to infer "none of the above".
+
+## What to watch for
+
+Two things make a result look like a bad model when it is not:
+
+**Prompt length.** Past roughly 14k tokens this stack stops producing language
+and starts producing garbage or nothing, without any error — and a single such
+request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
+near 5k, but the 252 documents over 14000 characters push toward the boundary
+with their RAG neighbours. A run that starts scoring zeros mid-way is more
+likely this than the model.
+
+**Ground truth.** It is the current filing, and over ten years conventions
+change. A disagreement with an old document is not automatically an error. The
+comparison between engines is the durable result; the absolute numbers are
+indicative.

scripts/paperless-eval/eval.py

@@ -0,0 +1,390 @@
+#!/usr/bin/env python3
+"""Measure what Paperless suggests for documents whose labels a human already chose.
+
+Runs both engines against the same documents:
+
+    /api/documents/<id>/suggestions/      the built-in scikit-learn classifier
+    /api/documents/<id>/ai_suggestions/   the LLM path, with RAG
+
+WHAT THIS ANSWERS, AND WHAT AN EARLIER VERSION DID NOT
+
+The two engines are not alternatives — the classifier costs 0.6 s per document
+and keeps running whatever the LLM does. So the useful question is not "which
+scores higher" but "what does the LLM add where the classifier is wrong", and
+that cannot be recovered from two aggregate percentages: 74% and 72% could
+overlap almost entirely or barely at all, and the difference between those two
+worlds is the entire value of running an LLM.
+
+So every document is written out individually, with both engines' verdicts, and
+the summary is derived from that rather than counted along the way.
+
+SCORING
+
+Correspondent and document type are each scored twice:
+
+    offered      the true label's id came back in `correspondents` /
+                 `document_types` — what the UI actually presents
+    recognised   also counting the true label's name appearing wordwise in
+                 `suggested_correspondents` / `suggested_document_types`
+
+The gap between them is one specific failure: the model names the right entity
+but paperless_ai.matching cannot resolve it, because difflib runs at
+MATCH_THRESHOLD = 0.8 and a correct but more verbose answer falls below it —
+"Generali Deutschland Krankenversicherung AG" against the existing "Generali
+AG". Reporting only the first blames the model for a matcher problem; only the
+second claims a usefulness the UI does not deliver.
+
+Document type is additionally split on whether the true type is "Sonstige",
+which holds 1133 of ~3900 documents and is the one label with no semantic
+content — a restaurant reservation lands there because nothing else fits. The
+classifier can memorise that; a model reading the document has to infer "none
+of the above". Scored separately rather than dropped, because the model does
+sometimes get it right.
+
+GROUND TRUTH
+
+The current filing, which deserves care: over ten years conventions change, and
+a disagreement with an old document is not automatically an error. Treat the
+comparison between engines as the result and the absolute numbers as
+indicative.
+
+Env:
+    PAPERLESS_URL     default http://127.0.0.1:8000
+    PGHOST/PGDATABASE standard libpq, for ground truth and the API token
+    EVAL_SAMPLE       documents to score, default 50
+    EVAL_ENGINES      comma-separated subset of classic,ai — default both
+    EVAL_LABEL        free-text name for this configuration, recorded in the
+                      output so results from different jobs stay distinguishable
+    EVAL_OUT          directory for results, default /results
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import secrets
+import sys
+import time
+import urllib.error
+import urllib.request
+
+import psycopg
+
+PAPERLESS_URL = os.environ.get("PAPERLESS_URL", "http://127.0.0.1:8000")
+SAMPLE = int(os.environ.get("EVAL_SAMPLE", "50"))
+ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
+LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
+OUT_DIR = os.environ.get("EVAL_OUT", "/results")
+
+# A backstop, not the real bound. Generation is capped server-side by
+# --n-predict on the llama-server sidecar, which is what actually stops a model
+# that will not terminate; this catches everything else — a wedged sidecar, a
+# request that never reaches it.
+#
+# 300 s against the ~73 s a healthy document takes end to end, prompt included.
+# It does work: llama-server logs `stop: cancel task` on the disconnect and
+# starts the next request 0.4 s later.
+REQUEST_TIMEOUT = 300
+
+CATCHALL_TYPE = "Sonstige"
+
+
+def log(msg: str) -> None:
+    print(f"[eval] {msg}", flush=True)
+
+
+def api_token(conn) -> str:
+    """A token for this run, created in the database.
+
+    The API needs one and this database is a disposable clone, so the run
+    issues its own against a superuser rather than depending on a token that
+    happened to come across in the restore.
+    """
+    token = secrets.token_hex(20)
+    with conn.cursor() as cur:
+        cur.execute(
+            "select id from auth_user where is_superuser and is_active order by id limit 1",
+        )
+        row = cur.fetchone()
+        if row is None:
+            raise SystemExit("no active superuser in the restored database")
+        user_id = row[0]
+        cur.execute("delete from authtoken_token where user_id = %s", (user_id,))
+        cur.execute(
+            "insert into authtoken_token (key, created, user_id) values (%s, now(), %s)",
+            (token, user_id),
+        )
+    conn.commit()
+    return token
+
+
+def sample(conn) -> list[dict]:
+    """The documents to score, ordered by md5 of the id.
+
+    Deterministic, so every engine and every configuration sees the same
+    documents and results stay comparable across jobs. Only documents carrying
+    both labels are usable as ground truth.
+    """
+    with conn.cursor() as cur:
+        cur.execute(
+            """
+            select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
+              from documents_document d
+              join documents_correspondent c on c.id = d.correspondent_id
+              join documents_documenttype  t on t.id = d.document_type_id
+             where d.correspondent_id is not null
+               and d.document_type_id is not null
+               and length(d.content) > 400
+             order by md5(d.id::text)
+             limit %s
+            """,
+            (SAMPLE,),
+        )
+        return [
+            {
+                "id": r[0],
+                "correspondent_id": r[1],
+                "document_type_id": r[2],
+                "correspondent_name": r[3],
+                "document_type_name": r[4],
+            }
+            for r in cur.fetchall()
+        ]
+
+
+def floor(conn) -> float:
+    """Always guessing the most frequent correspondent.
+
+    Without it a number like "70% correct" cannot be read: one correspondent
+    holds several hundred of ~3900 documents.
+    """
+    with conn.cursor() as cur:
+        cur.execute(
+            """
+            select max(c), sum(c) from (
+                select count(*) c from documents_document
+                 where correspondent_id is not null group by correspondent_id
+            ) x
+            """,
+        )
+        top, total = cur.fetchone()
+    # float() on both: psycopg returns Postgres numerics as Decimal, and
+    # mixing that with a float literal raises TypeError rather than coercing.
+    return 100.0 * float(top) / float(total) if total else 0.0
+
+
+def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
+    req = urllib.request.Request(
+        f"{PAPERLESS_URL}{path}",
+        headers={"Authorization": f"Token {token}"},
+    )
+    start = time.monotonic()
+    try:
+        with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT) as resp:
+            body = resp.read()
+        return json.loads(body), None, time.monotonic() - start
+    except urllib.error.HTTPError as err:
+        return None, f"HTTP {err.code}: {err.read()[:200]!r}", time.monotonic() - start
+    except Exception as err:  # noqa: BLE001 — a failed document is data, not a crash
+        return None, f"{type(err).__name__}: {err}", time.monotonic() - start
+
+
+def normalise(text: str) -> set[str]:
+    return {w for w in re.split(r"[^\w]+", text.lower()) if w}
+
+
+def recognised(payload: dict, field: str, want_name: str) -> bool:
+    """True when a suggested name contains every word of the true label's name.
+
+    Handles the observed failure directly: the model answers with a superset of
+    the stored name, which difflib rejects at 0.8 but a reader would accept.
+    """
+    want = normalise(want_name)
+    return any(want <= normalise(name) for name in payload.get(field) or [])
+
+
+def degenerate(payload: dict) -> str | None:
+    """Whether this looks like broken inference rather than a wrong guess.
+
+    Past roughly 14k prompt tokens this stack stops producing language — first
+    empty completions, then runs of digits and CJK characters — and one such
+    request poisons the llama.cpp slot until the pod restarts. Nothing in the
+    HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
+
+    Scored as a wrong answer it would silently depress a candidate's numbers,
+    and because the slot stays poisoned it would take the rest of the run with
+    it. Flagged instead, so the failure is visible as itself.
+    """
+    title = payload.get("title") or ""
+    if not title.strip():
+        return "empty title"
+    # Digits, punctuation and whitespace are normal in a document title —
+    # "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
+    # earlier version of this check. What broken inference produces is
+    # characters from other scripts, so look for those directly rather than
+    # inferring them from a shortage of Latin ones.
+    foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
+    if foreign > 0.2 * len(title):
+        return f"non-latin title: {title[:40]!r}"
+    return None
+
+
+def score_document(payload: dict, doc: dict) -> dict:
+    offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
+    offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
+    return {
+        "correspondent_offered": offered_corr,
+        "correspondent_recognised": offered_corr
+        or recognised(payload, "suggested_correspondents", doc["correspondent_name"]),
+        "type_offered": offered_type,
+        "type_recognised": offered_type
+        or recognised(payload, "suggested_document_types", doc["document_type_name"]),
+        "suggested_correspondents": payload.get("suggested_correspondents") or [],
+        "suggested_document_types": payload.get("suggested_document_types") or [],
+        "title": payload.get("title"),
+    }
+
+
+def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
+    log(f"=== {engine}: {len(docs)} documents")
+    rows = []
+    for n, doc in enumerate(docs, 1):
+        payload, error, elapsed = fetch(
+            f"/api/documents/{doc['id']}/{suffix}",
+            token,
+        )
+        row = {
+            "engine": engine,
+            "document_id": doc["id"],
+            "seconds": round(elapsed, 1),
+            "truth_correspondent": doc["correspondent_name"],
+            "truth_type": doc["document_type_name"],
+            "catchall_type": doc["document_type_name"] == CATCHALL_TYPE,
+        }
+        if payload is None:
+            row["error"] = error
+            log(f"  {doc['id']}: {error}")
+        else:
+            row.update(score_document(payload, doc))
+            # Only the AI path returns a title; the classifier answers with ids
+            # alone, so an "empty title" there is normal rather than broken.
+            if engine == "ai" and (broken := degenerate(payload)) is not None:
+                row["degenerate"] = broken
+                log(f"  {doc['id']}: DEGENERATE — {broken}")
+        rows.append(row)
+        if n % 10 == 0:
+            log(f"  {n}/{len(docs)}")
+    return rows
+
+
+def pct(hits: int, total: int) -> float:
+    return round(100.0 * hits / total, 1) if total else 0.0
+
+
+def summarise(rows: list[dict], engine: str) -> dict:
+    mine = [r for r in rows if r["engine"] == engine]
+    # Degenerate rows are excluded from the accuracy figures: they measure the
+    # backend giving up, not the model choosing wrongly, and leaving them in
+    # would quietly punish whichever candidate happened to trip the slot.
+    scored = [r for r in mine if "error" not in r and "degenerate" not in r]
+    failed = [r for r in mine if "error" in r]
+    broken = [r for r in mine if "degenerate" in r]
+    specific = [r for r in scored if not r["catchall_type"]]
+    catchall = [r for r in scored if r["catchall_type"]]
+    return {
+        "engine": engine,
+        "scored": len(scored),
+        "failed": len(failed),
+        # Non-zero means the run is suspect from that point on, not that the
+        # model is bad — the slot stays poisoned until the pod restarts.
+        "degenerate": len(broken),
+        "seconds_total": round(sum(r["seconds"] for r in mine)),
+        "correspondent_offered": pct(
+            sum(r["correspondent_offered"] for r in scored), len(scored)
+        ),
+        "correspondent_recognised": pct(
+            sum(r["correspondent_recognised"] for r in scored), len(scored)
+        ),
+        "type_offered": pct(sum(r["type_offered"] for r in scored), len(scored)),
+        "type_specific": pct(
+            sum(r["type_offered"] for r in specific), len(specific)
+        ),
+        "type_catchall": pct(
+            sum(r["type_offered"] for r in catchall), len(catchall)
+        ),
+    }
+
+
+def complement(rows: list[dict]) -> dict | None:
+    """What the LLM adds where the classifier is wrong, and what it costs.
+
+    The reason this script writes per-document rows at all. Only computed when
+    both engines ran, and only over documents both actually scored.
+    """
+    by_engine = {"classic": {}, "ai": {}}
+    for row in rows:
+        usable = "error" not in row and "degenerate" not in row
+        if usable and row["engine"] in by_engine:
+            by_engine[row["engine"]][row["document_id"]] = row
+    both = set(by_engine["classic"]) & set(by_engine["ai"])
+    if not both or not by_engine["ai"]:
+        return None
+
+    rescued = missed = broken = agreed = 0
+    for doc_id in both:
+        classic_hit = by_engine["classic"][doc_id]["correspondent_offered"]
+        ai_hit = by_engine["ai"][doc_id]["correspondent_offered"]
+        if classic_hit and ai_hit:
+            agreed += 1
+        elif not classic_hit and ai_hit:
+            rescued += 1
+        elif classic_hit and not ai_hit:
+            broken += 1
+        else:
+            missed += 1
+    return {
+        "documents": len(both),
+        "both_right": agreed,
+        "llm_rescued": rescued,
+        "llm_only_wrong": broken,
+        "both_wrong": missed,
+    }
+
+
+def main() -> int:
+    os.makedirs(OUT_DIR, exist_ok=True)
+    with psycopg.connect() as conn:
+        token = api_token(conn)
+        docs = sample(conn)
+        guess_floor = floor(conn)
+    log(f"label={LABEL} documents={len(docs)} floor={guess_floor:.1f}%")
+
+    rows: list[dict] = []
+    for engine, suffix in (("classic", "suggestions/"), ("ai", "ai_suggestions/")):
+        if engine in ENGINES:
+            rows.extend(run(engine, suffix, docs, token))
+
+    result = {
+        "label": LABEL,
+        "sample": len(docs),
+        "floor": round(guess_floor, 1),
+        "summaries": [summarise(rows, e) for e in ENGINES if e in ("classic", "ai")],
+        "complement": complement(rows),
+    }
+
+    with open(f"{OUT_DIR}/{LABEL}.jsonl", "w") as handle:
+        for row in rows:
+            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
+    with open(f"{OUT_DIR}/{LABEL}.summary.json", "w") as handle:
+        json.dump(result, handle, ensure_ascii=False, indent=2)
+
+    # Also to stdout, so the numbers are in the pod log and reachable via Loki
+    # without mounting the volume.
+    log(json.dumps(result, ensure_ascii=False, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/paperless-eval/job-qwen35.yaml

@@ -0,0 +1,324 @@
+---
+# One configuration under test, complete in one pod.
+#
+# A paperless instance can only point at one chat endpoint, so measuring two
+# candidates at once means two paperless instances — which makes the pod, not
+# the namespace, the right unit. Everything a measurement needs is here except
+# the database and the embedding endpoint, both of which are read-only and
+# shared.
+#
+# The three services are native sidecars (initContainers with
+# restartPolicy: Always), not regular containers. That matters twice: they
+# start in order and each must pass its probes before the next begins, which
+# is what keeps paperless from racing the database; and when the eval
+# container exits they are torn down, so the Job actually completes instead of
+# running forever behind a llama-server that never returns.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: eval-qwen35
+  namespace: paperless-eval
+  labels:
+    app.kubernetes.io/name: paperless-eval
+    app.kubernetes.io/component: eval
+spec:
+  # A failed run is re-read, not retried blind: a retry would spend another
+  # hour reproducing whatever went wrong.
+  backoffLimit: 0
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless-eval
+        app.kubernetes.io/component: eval
+    spec:
+      restartPolicy: Never
+      enableServiceLinks: false
+      # One measurement per node. Two jobs sharing a node would share its iGPU,
+      # and every timing in the result would then describe contention rather
+      # than the model — which is exactly how a decode rate of 4.7 tokens/s got
+      # attributed to context length once already.
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  app.kubernetes.io/component: eval
+              topologyKey: kubernetes.io/hostname
+      securityContext:
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        # A private copy of /data — the search index and, the expensive part,
+        # the LLM index the RAG path needs.
+        #
+        # It cannot be mounted live and shared: LLM_INDEX_DIR is hardcoded to
+        # DATA_DIR / "llm_index", the store behind it is SQLite, and SQLite
+        # locking across pods over CephFS is not dependable. Nor can it be left
+        # out — an empty llm_index makes retrieve_similar_nodes return nothing,
+        # and the job would quietly measure the no-RAG path while reporting a
+        # RAG result.
+        #
+        # ~160 MiB, so seconds. A regular initContainer: it must finish before
+        # the sidecars start.
+        - name: seed-data
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              # -r, not -a: preserving ownership needs CHOWN, which this
+              # container does not have, and cp then exits non-zero after
+              # copying the files anyway. Running as 1000 makes the copy land
+              # with the ownership paperless needs — it writes to the LLM index
+              # while serving.
+              cp -r /golden/. /data/
+              du -sh /data/index /data/llm_index
+              # Refuse to start rather than measure the wrong thing. An empty
+              # llm_index makes retrieve_similar_nodes return nothing, and the
+              # run would report a RAG result having never retrieved anything —
+              # which is exactly what the first attempt at this job did, because
+              # the golden volume had not been populated yet. The index is
+              # ~125 MiB once built; anything under 1 MiB is not one.
+              bytes=$(du -sb /data/llm_index | cut -f1)
+              if [ "$bytes" -lt 1000000 ]; then
+                echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
+                echo "smoke-test deployment first:" >&2
+                echo "  kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
+                echo "    python manage.py document_llmindex rebuild" >&2
+                exit 1
+              fi
+          volumeMounts:
+            - name: golden
+              mountPath: /golden
+              readOnly: true
+            - name: data
+              mountPath: /data
+          securityContext:
+            runAsUser: 1000
+            runAsGroup: 1000
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+        - name: llama-server
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          restartPolicy: Always
+          args:
+            - --hf-repo
+            - ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
+            - --alias
+            - qwen3.6-35b-a3b
+            - --n-gpu-layers
+            - "99"
+            - --threads
+            - "4"
+            # The production ceiling. Not because the flag value itself breaks
+            # anything — bisection showed the corruption boundary is prompt
+            # length, around 14000 tokens, whatever --ctx-size says — but
+            # because measuring against a different ceiling than production
+            # runs would not describe production.
+            - --ctx-size
+            - "16384"
+            - --parallel
+            - "1"
+            - --ubatch-size
+            - "1024"
+            # The default is -1, infinity: nothing on the server caps how long
+            # a request generates, and paperless sets no budget of its own and
+            # waits. A classification that needs ~300 tokens ran to 10269 at a
+            # healthy 11 tok/s, and three of the first eleven documents did it
+            # — 900 s each, most of an hour spent on four documents.
+            #
+            # 768 is generous against the ~300 a correct answer takes, and a
+            # truncated response is better evidence than a timeout: it shows
+            # what the model was doing instead of stopping.
+            - --n-predict
+            - "768"
+            # Without this, Qwen3.6 spends the response on reasoning_content
+            # and returns empty content — 983 tokens into a classification that
+            # needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
+            # slow or broken model and is neither.
+            #
+            # A candidate whose template has no such switch has to be measured
+            # as it comes, and that is a fair part of its result: paperless
+            # gives the model no token budget and waits.
+            - --chat-template-kwargs
+            - '{"enable_thinking": false}'
+            - --no-webui
+          env:
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: llm
+              containerPort: 8080
+          volumeMounts:
+            - name: models
+              mountPath: /models
+          # ~20 GB of weights off CephFS, so a generous budget: 60 * 20s.
+          startupProbe:
+            httpGet:
+              path: /health
+              port: llm
+            periodSeconds: 20
+            failureThreshold: 60
+          resources:
+            requests:
+              cpu: "1"
+              memory: 24Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              memory: 32Gi
+              gpu.intel.com/i915: "1"
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+        - name: redis
+          # Its own, not the namespace's: two paperless instances sharing a
+          # celery broker would take each other's tasks.
+          image: redis:8.10.1-alpine
+          restartPolicy: Always
+          args: ["redis-server", "--save", "", "--loglevel", "warning"]
+          startupProbe:
+            tcpSocket:
+              port: 6379
+            periodSeconds: 2
+            failureThreshold: 30
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+        - name: paperless
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
+          restartPolicy: Always
+          envFrom:
+            - configMapRef:
+                name: paperless-eval-config
+          env:
+            # Overrides the namespace ConfigMap: this pod's own llama-server,
+            # and its own redis rather than the shared Service.
+            - name: PAPERLESS_AI_LLM_ENDPOINT
+              value: http://127.0.0.1:8080/v1
+            - name: PAPERLESS_AI_LLM_MODEL
+              value: qwen3.6-35b-a3b
+            - name: PAPERLESS_REDIS
+              value: redis://127.0.0.1:6379
+            - name: PAPERLESS_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-eval-secrets
+                  key: secret-key
+            - name: PAPERLESS_DBUSER
+              valueFrom:
+                secretKeyRef:
+                  name: pprleval-pg-app
+                  key: username
+            - name: PAPERLESS_DBPASS
+              valueFrom:
+                secretKeyRef:
+                  name: pprleval-pg-app
+                  key: password
+          ports:
+            - name: http
+              containerPort: 8000
+          volumeMounts:
+            - name: data
+              mountPath: /usr/src/paperless/data
+            - name: media
+              mountPath: /usr/src/paperless/media
+            - name: consume
+              mountPath: /usr/src/paperless/consume
+          startupProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            periodSeconds: 20
+            failureThreshold: 180
+          # Same as production, and required: the entrypoint starts as root,
+          # chowns the volumes and drops to uid 1000 via setgid/setuid.
+          securityContext:
+            runAsNonRoot: false
+            allowPrivilegeEscalation: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              memory: 2Gi
+      containers:
+        - name: eval
+          # The paperless image again, for psycopg and a python that matches
+          # the application — not because the script needs paperless itself.
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
+          command: ["python3", "/eval/eval.py"]
+          env:
+            - name: EVAL_LABEL
+              value: qwen35
+            - name: EVAL_SAMPLE
+              value: "50"
+            - name: PGHOST
+              value: pprleval-pg-rw
+            - name: PGDATABASE
+              value: paperless
+            - name: PGUSER
+              valueFrom:
+                secretKeyRef:
+                  name: pprleval-pg-app
+                  key: username
+            - name: PGPASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: pprleval-pg-app
+                  key: password
+          volumeMounts:
+            - name: script
+              mountPath: /eval
+            - name: results
+              mountPath: /results
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 512Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-models
+        - name: results
+          persistentVolumeClaim:
+            claimName: paperless-eval-results
+        - name: script
+          configMap:
+            name: paperless-eval-script
+        # The namespace's /data, mounted read-only and copied out by seed-data
+        # above. Never mounted as the live /data of a job.
+        - name: golden
+          persistentVolumeClaim:
+            claimName: paperless-eval-golden
+            readOnly: true
+        # paperless insists on all three paths existing. media and consume stay
+        # empty — nothing is consumed during a measurement — while data is
+        # seeded with the copy.
+        - name: data
+          emptyDir: {}
+        - name: media
+          emptyDir: {}
+        - name: consume
+          emptyDir: {}