fix(paperless-eval): score a right answer the matcher threw away

The AI engine reports an unresolved guess as a name in `suggested_correspondents` rather than an id in `correspondents`, and paperless_ai.matching only resolves it when difflib.get_close_matches clears MATCH_THRESHOLD = 0.8. A correct but more verbose answer does not: "Generali Deutschland Krankenversicherung AG" fails against the existing "Generali AG", leaving `correspondents` empty. Scored as it was, that counts as a miss, which would have blamed the model for a matcher limit. Score both instead — the id as offered, and the name matched wordwise against the true label — so the two causes stay separable. The classic engine emits no `suggested_*` fields, so its two columns coincide and the comparison stays fair either way. Names now come out of postgres alongside the ids, tab-separated because '|' occurs in correspondent names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(scripts): end-to-end evaluation of Paperless suggestions
2026-09-02 14:05:00 +02:00 · 2026-09-02 13:32:33 +02:00
30 changed files with 242 additions and 2020 deletions
@@ -10,18 +10,6 @@ when:
      # install-netbird-proxy.yml also applies the edge bouncer.
      - "ansible/roles/crowdsec-edge-bouncer/**"
      - "ansible/plays/install-netbird-proxy.yml"
-      # The variables that steer those roles, not just the roles themselves.
-      # Without these two, a change that only touches inventory never deploys
-      # and nothing says so: arming the bouncer lives in host_vars, and its
-      # LAPI key lives in group_vars, so both would sit in git looking applied
-      # while the hosts kept running the previous configuration.
-      #
-      # pve_hosts.yml is deliberately NOT listed. It is the single inventory
-      # file for the whole fleet, so listing it would redeploy the proxies for
-      # any unrelated host edit. The cost is that a change to [netbird_proxy]
-      # group membership needs a manual deployment run.
-      - "ansible/inventory/host_vars/proxy-*.yml"
-      - "ansible/inventory/group_vars/netbird_proxy/**"
      - ".woodpecker/deploy-netbird-proxy.yml"
  - event: deployment
    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
@@ -28,23 +28,14 @@ crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25
 crowdsec_bouncer_api_url: https://crowdsec.renner.uno/
 crowdsec_bouncer_api_key: "{{ vault_crowdsec_edge_bouncer_key }}"

-# Enforcing. The bouncer creates two nftables tables (ip crowdsec, ip6
-# crowdsec6) holding the decisions it pulled, touches no existing chain and
-# sets no chain policy — SSH and :443 stay open unless the source is banned.
+# STARTS INERT, and that is the point.
 #
-# `dry-run` is the other value: it pulls and logs what it would block without
-# creating a single rule. That is what a new host should be given before it
-# enforces, and what an existing one can be dropped back to, per host through
-# inventory/host_vars, without disarming the rest.
-#
-# Worth knowing before arming anything new: the first bouncer on a path
-# enforces the whole CAPI set at once — around 15k addresses today — so the
-# false-positive window belongs before the switch, not after it. And because
-# this chain sits on the INPUT hook while these hosts run no other firewall,
-# there is no conntrack rule accepting established flows ahead of it: a banned
-# address is dropped even when it is answering our own outbound connection.
-# That is why capi_whitelists.yaml carries the proxies and netbird.renner.uno.
-crowdsec_bouncer_mode: nftables
+# dry-run pulls decisions and logs what it would block without creating a
+# single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI
+# decisions at once on the only external entry path, so the false-positive
+# window has to come before the enforcement, not after it. Flip to `nftables`
+# one host at a time once the log looks right.
+crowdsec_bouncer_mode: dry-run

 crowdsec_bouncer_update_frequency: 10s
 crowdsec_bouncer_log_level: info
@@ -19,40 +19,13 @@
    state: directory
    mode: "0755"

-# Fetched on the CONTROLLER and pushed, not pulled by the host.
-#
-# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
-# publishes an AAAA record — a get_url on the host fails with "Network is
-# unreachable" and no amount of retrying fixes it. Mirroring into
-# gitea.renner.uno is no help either: its public records are the internal LB
-# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
-# upstream ships no container image for this bouncer, which is expected —
-# it manipulates host netfilter.
-#
-# The CI runner has v4, so it does the fetching. The checksum is verified
-# there, before anything is copied to a host.
- name: Fetch the pinned bouncer release on the controller
+- name: Fetch the pinned bouncer release
  ansible.builtin.get_url:
    url: >-
      https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
      crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
-    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
-    mode: "0644"
-  delegate_to: localhost
-  become: false
-  run_once: true
-
-# Only the download had to move to the controller — the extraction stays on
-# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
-# tar type None. GNU tar required"), while both host distributions have GNU
-# tar. Shipping the archive rather than the unpacked binary also keeps the
-# artefact that the checksum was verified against intact all the way to the
-# host.
- name: Copy the bouncer archive to the host
-  ansible.builtin.copy:
-    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
    mode: "0644"

 # Only the binary. The tarball also carries install.sh / upgrade.sh /
@@ -95,7 +95,7 @@ spec:
              memory: 32Mi
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.9.0 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -75,62 +75,11 @@ spec:
            # many small per-expert matmuls that sets the pace, not bandwidth.
            - --threads
            - "4"
-            # 16k is room to spare, not a safety limit. The limit that matters
-            # is not set here at all.
-            #
-            # Isolated by bisection on a scratch pod: this stack breaks on
-            # PROMPT length, and the value of --ctx-size makes no difference to
-            # it. Measured at --ctx-size 65536 with a single slot, so with four
-            # times the headroom:
-            #
-            #     13588 prompt tokens   coherent answer
-            #     15086                 one token, immediate EOS, empty content
-            #     16088                 CJK characters and digit runs
-            #
-            # A single request past that boundary then poisons the slot: the
-            # same 13588-token prompt that answered cleanly a minute earlier
-            # returns one empty token afterwards, and stays broken until the
-            # pod restarts. That is why it looked intermittent, and why a
-            # 650-token classification once came back as 1025 tokens of "!" —
-            # it followed Home Assistant's ~19k prompts through the same slot.
-            #
-            # Nothing reports any of it: finish_reason "stop", a well-formed
-            # OpenAI response, HTTP 200, /health green.
-            #
-            # So the operating rule is about callers, not this flag. Paperless
-            # sits near 5k and is safe; Home Assistant's Assist prompt measured
-            # 18910 and never had a chance. An earlier version of this comment
-            # blamed --ctx-size 32768, which the bisection disproves — 32768
-            # and 65536 both answer correctly as long as the prompt stays under
-            # the boundary.
-            #
-            # The same stack also fails hard at --ubatch-size 2048, which is
-            # the loud version of a limit this one keeps quiet about.
-            #
+            # 16k rather than 8k because the measured prefill falloff is mild
+            # and this model's KV cache is unusually cheap — only 10 of its 40
+            # layers use full attention.
            - --ctx-size
            - "16384"
-            # One slot, so the 16384 belongs to one request at a time.
-            #
-            # Left on auto, llama.cpp opens four slots and — because auto is
-            # also what enables --kv-unified — has them share a single buffer
-            # of this size. Four Paperless requests at once wanted 6209, 6472,
-            # 4303 and 1271 tokens against those 16384, and all four died with
-            # "Context size has been exceeded".
-            #
-            # They were concurrent because a client timeout does not stop the
-            # generation behind it: each abandoned request keeps consuming the
-            # shared budget the next one needs, so one slow document is enough
-            # to start the pile-up. Queueing is the behaviour worth having
-            # here — nothing calls this server in parallel on purpose, and
-            # Paperless's celery worker runs at concurrency 1.
-            #
-            # No liveness probe change goes with this. The server recovered on
-            # its own once the batch drained, so a probe that restarted the pod
-            # would have made the incident worse; with a single slot, one that
-            # asked for a real completion would also queue behind live work and
-            # kill healthy pods during a long document.
-            - --parallel
-            - "1"
            # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
            # with ErrorDeviceLost.
            - --ubatch-size
@@ -41,7 +41,7 @@ spec:
          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
          args:
            - --hf-repo
-            - gpustack/bge-m3-GGUF:Q8_0
+            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
            # Serves POST /v1/embeddings.
            - --embeddings
            # A short, stable id for consumers. Without it the model is
@@ -50,41 +50,28 @@ spec:
            # One name only: llama-server keeps aliases in a sorted set and
            # reports the first as the id, so a second name can steal it.
            - --alias
-            - bge-m3
-            # bge-m3 is trained for CLS pooling — a different type from the
-            # Qwen family this replaces, which uses last-token. Left unspecified
-            # llama-server takes the GGUF default, which is not visible from
-            # outside and would silently produce vectors the model was not
-            # trained to emit. Read from the model's own 1_Pooling/config.json
-            # rather than assumed; of the candidates measured, three different
-            # pooling types were needed.
+            - qwen3-embedding-0.6b
+            # Qwen3-Embedding is trained for last-token pooling. Left
+            # unspecified llama-server takes the GGUF's default, which is not
+            # visible from outside and would silently produce vectors the model
+            # was not trained to emit. Set before anything is indexed: changing
+            # pooling later invalidates every stored embedding.
            - --pooling
-            - cls
+            - last
            - --n-gpu-layers
            - "99"
            - --threads
            - "2"
-            # Paperless chunks at 1024 tokens, but counts them with its own
-            # tokenizer while this server measures the batch in bge-m3's.
-            # Those disagree: XLM-RoBERTa splits German into more pieces than
-            # llama_index's counter does, so a chunk Paperless considers full
-            # at 1024 arrived here as 1069 and the request failed outright —
-            # "input (1069 tokens) is too large to process".
-            #
-            # The headroom therefore has to absorb a tokenizer ratio, not just
-            # a rounding error, and 1.5x covers far more than the 1.044 that
-            # broke it. The defaults are batch 2048 / ubatch 512, and with
-            # --embeddings llama-server clamps both to the smaller rather than
-            # run them mismatched, so both are set.
-            #
-            # 2048 stays avoided: that is where the chat model met
-            # ErrorDeviceLost on this Vulkan stack. That failure came under a
-            # 20 GB model's memory pressure and may not bind a 600 MB one, but
-            # this does not need to find out.
+            # Paperless chunks documents at 1024 tokens. The defaults are
+            # batch 2048 / ubatch 512, and with --embeddings llama-server
+            # clamps both down to 512 rather than run them mismatched — below
+            # the chunk size. Matching both to 1024 fits a whole chunk in one
+            # pass; 2048 is deliberately avoided, since that is where this
+            # Vulkan stack fails with ErrorDeviceLost.
            - --batch-size
-            - "1536"
+            - "1024"
            - --ubatch-size
-            - "1536"
+            - "1024"
            # Room for a full chunk plus overhead rather than exactly one.
            - --ctx-size
            - "4096"
@@ -104,14 +91,6 @@ spec:
            capabilities:
              drop:
                - ALL
-          # Measured, not estimated: embedding 400 real documents peaked at
-          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
-          # well above that because the footprint is dominated by the four slots
-          # at 4096 context and the Vulkan working set rather than by the
-          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
-          # same settings for the same 600 MiB of weights, and several
-          # candidates were OOMKilled at limits that looked generous against
-          # their model size.
          resources:
            requests:
              cpu: 250m
@@ -119,7 +98,7 @@ spec:
              gpu.intel.com/i915: "1"
            limits:
              cpu: "2"
-              memory: 8Gi
+              memory: 4Gi
              gpu.intel.com/i915: "1"
          # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
          # model needs, but still sized for a cold download.
@@ -1,36 +0,0 @@
-# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
-# returned zero series and Prometheus held no cs_* metric at all, so the
-# detection side of the ingress path was invisible — including whether the
-# agents were reading logs and whether any bouncer was pulling decisions.
-#
-# One monitor for both services. They carry the same shape (a `metrics` port on
-# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
-# a single object rather than two near-identical ones:
-#
-#   crowdsec-service        the LAPI      — decisions, bouncer + machine requests
-#   crowdsec-agent-service  the agents    — acquisition, parsers, scenarios
-#
-# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
-# without a port restriction, so no policy change is needed.
-#
-# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: crowdsec
-  namespace: crowdsec
-  labels:
-    app.kubernetes.io/name: crowdsec
-    release: kube-prometheus-stack
-spec:
-  selector:
-    matchExpressions:
-      - key: app
-        operator: In
-        values:
-          - crowdsec-service
-          - crowdsec-agent-service
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 30s
@@ -23,7 +23,6 @@ resources:
  - prometheusrule-cnpg-backup.yaml
  - alloy-servicemonitor.yaml
  - cnpg-podmonitors.yaml
-  - crowdsec-servicemonitor.yaml
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
@@ -1,163 +0,0 @@
-{
-  "annotations": {"list": []},
-  "description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "panels": [
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
-      "id": 1,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
-      "title": "Aktive Decisions",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
-      "id": 2,
-      "options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
-      "title": "Bouncer, die abholen",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
-      "id": 3,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
-      "title": "Gelesene Logzeilen/s",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
-      "id": 4,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
-      "title": "Lokale Overflows (24 h)",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
-      "id": 5,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
-      "title": "Bouncer-Abholrate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
-      "id": 6,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
-      "title": "Aktive Decisions nach Grund",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
-      "id": 7,
-      "options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
-      "title": "Szenario-Overflows (je Stunde)",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
-      "id": 8,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
-      ],
-      "title": "Parser",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
-      "id": 9,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
-      "title": "Whitelist-Treffer",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
-      "id": 10,
-      "options": {
-        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
-        "content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
-      },
-      "title": "Blinder Fleck",
-      "type": "text"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["homelab", "crowdsec", "security"],
-  "templating": {"list": []},
-  "time": {"from": "now-6h", "to": "now"},
-  "timepicker": {},
-  "timezone": "",
-  "title": "CrowdSec Overview",
-  "uid": "crowdsec-overview",
-  "version": 1,
-  "weekStart": ""
-}
@@ -38,9 +38,6 @@ configMapGenerator:
  - name: dashboard-blackbox-overview
    files:
      - blackbox-overview.json
-  - name: dashboard-crowdsec-overview
-    files:
-      - crowdsec-overview.json
  - name: dashboard-disk-health
    files:
      - disk-health.json
@@ -1,116 +0,0 @@
---
-# A read-only view of paperless's backup store, used as a recovery source and
-# nothing else.
-#
-# It points at the production prefix, which is the whole point — the eval
-# instance is a copy of the real archive's metadata — and that makes what is
-# ABSENT here the safety-critical part:
-#
-#   no retentionPolicy   The production store has "30d". Without one here
-#                        there is no policy to enforce, and enforcement is not
-#                        tied to taking a backup: the plugin runs it from the
-#                        instance sidecar on a timer
-#                        (instanceSidecarConfiguration.retentionPolicyIntervalSeconds,
-#                        default 1800). So "this cluster never backs up" would
-#                        not have been enough on its own.
-#
-# A second property makes it belt-and-braces. Retention runs
-# barman-cloud-backup-delete, which takes a server name and works inside
-# <destinationPath>/<serverName> — the store's own status keys its recovery
-# window by "pprl-pg", confirming that layout. This cluster is pprleval-pg, so
-# even an accidental retention policy would address a prefix that holds nothing
-# rather than production's backups.
-#
-# The cluster below completes the guarantee with isWALArchiver: false, so
-# nothing is written back either. It does load the plugin — that is required
-# for the restore to work at all — and read-only comes from the flag rather
-# than from leaving the plugin out.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: pprleval-pg-source
-  namespace: paperless-eval
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
---
-# A disposable clone of the paperless database, for measuring how well the
-# local LLM classifies documents without running the experiment against the
-# live archive.
-#
-# Losing this cluster costs a re-restore and nothing else, which is the intent
-# — it is rebuilt whenever a test needs a clean starting point.
-#
-# One instance, not two. Replication buys availability, and an eval instance
-# that is down is simply an eval that waits.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: pprleval-pg
-  namespace: paperless-eval
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  enableSuperuserAccess: true
-  storage:
-    # Delete-reclaim, unlike the Retain default: this volume is a copy that
-    # gets rebuilt, so it should vanish with the cluster rather than leave a
-    # Released PV and an orphaned RBD image behind (issue #262).
-    storageClass: ceph-block-ephemeral
-    size: 10Gi
-  # Declared with isWALArchiver: false, which is what kubernetes/RECOVERY.md
-  # arrived at after an earlier version of that runbook said to omit it: the
-  # instance manager loads this plugin to replay WAL during the restore, and
-  # without it the instance never starts ("Unknown plugin:
-  # barman-cloud.cloudnative-pg.io").
-  #
-  # externalClusters below only says WHERE the backup is; it does not get the
-  # plugin injected into the pod. So "no plugin" is not the way to keep this
-  # read-only — "plugin, not archiving" is, and it is the combination verified
-  # on im-pg on 2026-07-25.
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      enabled: true
-      isWALArchiver: false
-      parameters:
-        barmanObjectName: pprleval-pg-source
-  bootstrap:
-    recovery:
-      source: pprl-pg
-      database: paperless
-      owner: paperless
-  externalClusters:
-    - name: pprl-pg
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: pprleval-pg-source
-          serverName: pprl-pg
-  postgresql:
-    parameters:
-      shared_buffers: "256MB"
-      max_connections: "100"
-      # The production tuning for write reduction is deliberately not copied.
-      # It exists to spare the SSDs under a database that runs for years; this
-      # one is read-only for hours at a time and then deleted.
-  resources:
-    requests:
-      cpu: 200m
-      memory: 512Mi
-    limits:
-      memory: 1Gi
@@ -1,13 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: paperless-eval
-# The instance only. The measurement harness lives in scripts/paperless-eval/
-# and is applied by hand — a Job is immutable, re-running one means deleting and
-# recreating it, and Flux would just put the old one back. Nothing here should
-# converge on a finished measurement.
-resources:
-  - secrets.yaml
-  - cluster.yaml
-  - pvcs.yaml
-  - redis.yaml
-  - paperless.yaml
@@ -1,203 +0,0 @@
---
-# Deliberately much smaller than the production ConfigMap. Everything that
-# exists to serve people is gone: no remote-user auth, no CSRF origins, no
-# barcode or mail consumption, no Tika/Gotenberg. Nothing consumes documents
-# here, and nothing signs in.
-#
-# What remains is the database connection and the AI settings, because those
-# are what the evaluation measures.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: paperless-eval-config
-  namespace: paperless-eval
-data:
-  PAPERLESS_URL: "http://paperless.paperless-eval.svc.cluster.local"
-  PAPERLESS_ALLOWED_HOSTS: "*"
-  PAPERLESS_TIME_ZONE: "Europe/Berlin"
-  PAPERLESS_REDIS: "redis://redis:6379"
-  PAPERLESS_DBENGINE: "postgresql"
-  PAPERLESS_DBHOST: "pprleval-pg-rw"
-  PAPERLESS_DBNAME: "paperless"
-  # Polling off, so a pod that finds an empty consume volume does not spend a
-  # thread looking at it every ten seconds for the length of a run.
-  PAPERLESS_CONSUMER_POLLING_INTERVAL: "0"
-
-  # The AI settings under test. The endpoints are placeholders that each eval
-  # job overrides for the model it is measuring — a job runs its own
-  # llama-server beside paperless in the same pod, so the value that matters at
-  # run time is the one in the job, not the one here.
-  #
-  # Same caveat as production, and sharper here: paperless reads these from an
-  # ApplicationConfiguration row and falls back to the environment only when
-  # that row is empty. The row comes across in the restored database, so
-  # anything configured in the production admin UI arrives with it and silently
-  # wins over this ConfigMap — an eval would then measure production's settings
-  # while appearing to measure the job's.
-  #
-  # Check it before trusting any result, and after every fresh restore:
-  #
-  #   select llm_backend, llm_model, llm_embedding_backend, llm_embedding_model
-  #     from paperless_applicationconfiguration;
-  #
-  # Every column must be null for the environment to apply.
-  PAPERLESS_AI_ENABLED: "true"
-  PAPERLESS_AI_LLM_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_ENDPOINT: "http://127.0.0.1:8080/v1"
-  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
-  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
-  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
-  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "900"
-  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
-  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
-  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
---
-# A base instance that proves the restored database is usable and that the
-# search index and classifier build against it. The evaluation itself does not
-# run here — each job brings its own paperless container so it can point at its
-# own model — but this is what makes a broken clone visible before a job spends
-# hours on it.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: paperless
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: paperless
-  annotations:
-    configmap.reloader.stakater.com/reload: paperless-eval-config
-    # Both, like production: the secret refreshes on the same 10m cycle, and
-    # without this the pod would keep a rotated key until something else
-    # restarted it — which defeats the point of matching the interval.
-    secret.reloader.stakater.com/reload: paperless-eval-secrets
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: paperless
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless
-    spec:
-      enableServiceLinks: false
-      securityContext:
-        fsGroup: 1000
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: paperless
-          # No $imagepolicy marker, unlike apps/paperless: a measurement is
-          # only comparable against a fixed version, so this tag moves when
-          # somebody decides to re-measure, not when a release lands.
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          ports:
-            - name: http
-              containerPort: 8000
-          envFrom:
-            - configMapRef:
-                name: paperless-eval-config
-          env:
-            - name: PAPERLESS_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-eval-secrets
-                  key: secret-key
-            - name: PAPERLESS_DBUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PAPERLESS_DBPASS
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          volumeMounts:
-            - name: data
-              mountPath: /usr/src/paperless/data
-            - name: media
-              mountPath: /usr/src/paperless/media
-            - name: consume
-              mountPath: /usr/src/paperless/consume
-          # 60 minutes, against production's 10, and matching this app's Flux
-          # Kustomization timeout.
-          #
-          # Production's budget was sized for a from-scratch boot on this same
-          # corpus and held, so 10 would probably do — but there the volume is
-          # Retain and the index survives, making that a one-off. Here the
-          # volume is ephemeral and the index is rebuilt on every re-test, so
-          # the expensive path is the normal one. A killed pod starts the work
-          # over rather than resuming, which turns a slow boot into a
-          # crash-loop that stalls Flux behind wait: true.
-          #
-          # The LLM index is not in this budget: it is a celery task, and boot
-          # only runs init-llmindex-migrate, a migration check.
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            timeoutSeconds: 5
-            failureThreshold: 180
-          livenessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 120
-            periodSeconds: 30
-            timeoutSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 30
-            periodSeconds: 10
-            timeoutSeconds: 5
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              memory: 2Gi
-          # Production's context verbatim, and it has to be: the paperless-ngx
-          # entrypoint starts as root, chowns the mounted data/media/consume
-          # volumes and then drops to uid 1000 via setgid/setuid. Without these
-          # the same image fails that sequence with EPERM.
-          #
-          # It bites harder here than in production, where those volumes were
-          # chowned once years ago and keep their ownership. These are fresh on
-          # every rebuild, so the chown is not a no-op.
-          securityContext:
-            runAsNonRoot: false
-            allowPrivilegeEscalation: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-golden
-        - name: media
-          persistentVolumeClaim:
-            claimName: paperless-eval-media
-        - name: consume
-          persistentVolumeClaim:
-            claimName: paperless-eval-consume
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: paperless
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: paperless
-  ports:
-    - port: 80
-      targetPort: http
@@ -1,129 +0,0 @@
---
-# All four use ceph-block-ephemeral rather than the Retain default. Everything
-# here is derived — a restored database, an index built from it, a cache —  so
-# the volumes should go when the namespace does instead of leaving Released PVs
-# and orphaned RBD images to reclaim by hand (issue #262).
-#
-# /data holds the Tantivy search index and the classifier pickle, both rebuilt
-# from the database, and the LLM index — which is the expensive one at roughly
-# three hours for the full corpus.
-#
-# Measured on production for the same ~3900 documents: 36 MiB search index,
-# 125 MiB LLM index. 10Gi is far more than that needs, and deliberately so —
-# an embedding model with wider vectors than bge-m3's 1024 dimensions is
-# exactly what phase two varies, and running out of space mid-run would cost
-# the whole rebuild.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  # Named for its role rather than its mount point: the smoke-test deployment
-  # builds /data here, and every eval job copies it out. Replaces the earlier
-  # RWO paperless-eval-data, whose access mode could not be widened in place —
-  # a bound claim's spec is immutable.
-  name: paperless-eval-golden
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  # RWX, because the eval jobs read it too — each copies it into its own
-  # emptyDir rather than mounting it live. The vector store is SQLite, and
-  # SQLite locking across pods over CephFS is not something to rely on.
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem-ephemeral
-  resources:
-    requests:
-      storage: 10Gi
---
-# Empty, and stays that way. Paperless refuses to start without the path, but
-# the AI path reads title, content and filename out of the database and never
-# opens a file — so the documents themselves are not copied here. That is what
-# keeps this instance cheap and keeps a second copy of the archive off the
-# cluster.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-media
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
---
-# Also empty: nothing is consumed here, but the path has to exist.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-consume
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
---
-# The eval jobs' model cache, RWX so several jobs can read the same weights at
-# once — that is the whole point of measuring two candidates in parallel.
-#
-# Separate from the ai namespace's llama-server-models despite holding some of
-# the same files: PVCs do not cross namespaces. A model therefore downloads
-# once here and is then shared by every job, which is the cost a new candidate
-# carries anyway.
-#
-# NOT ephemeral, unlike everything else in this namespace: re-downloading tens
-# of gigabytes on a slow uplink is a worse outcome than an orphaned volume.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: llama-models
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 120Gi
---
-# Results outlive the jobs that produce them: a Job is deleted and recreated to
-# re-run a configuration, and comparing configurations is the entire exercise.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-results
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 1Gi
---
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-redis
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
@@ -1,81 +0,0 @@
---
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: redis
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: redis
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: redis
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: redis
-    spec:
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 999
-        fsGroup: 999
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: redis
-          # No $imagepolicy marker; see the note on the paperless image.
-          image: redis:8.10.1-alpine
-          # No --save: this holds the celery queue and the suggestion cache,
-          # both of which are regenerated. Persisting them across a restart of
-          # a throwaway instance would only preserve stale suggestions.
-          args:
-            - redis-server
-            - --save
-            - ""
-            - --loglevel
-            - warning
-          ports:
-            - name: redis
-              containerPort: 6379
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          livenessProbe:
-            tcpSocket:
-              port: redis
-            initialDelaySeconds: 10
-          readinessProbe:
-            exec:
-              command: ["redis-cli", "ping"]
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 20m
-              memory: 64Mi
-            limits:
-              memory: 256Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-redis
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: redis
-  ports:
-    - port: 6379
-      targetPort: redis
@@ -1,45 +0,0 @@
---
-# The same Hetzner S3 key every CNPG-using namespace pulls, needed here only so
-# the recovery source can be read. See apps/paperless for why one shared key
-# rather than per-app paths.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: paperless-eval
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: infrastructure/hetzner-s3
---
-# Django's secret key, shared with the production instance rather than given
-# its own vault entry.
-#
-# It signs sessions and CSRF tokens, and this instance has no ingress and no
-# users — nothing signs in, so nothing is at stake in sharing it. A separate
-# entry would mean a vault write before the namespace can start, for a
-# throwaway instance that gets rebuilt.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: paperless-eval-secrets
-  namespace: paperless-eval
-spec:
-  # 10m, matching apps/paperless rather than the 1h default: the point of
-  # sharing the key is that the two hold the same value, and a slower refresh
-  # here would let them diverge for up to an hour after a rotation.
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: paperless-eval-secrets
-  dataFrom:
-    - extract:
-        key: apps/paperless
@@ -82,10 +82,7 @@ data:
  # primarily English archives — the better fit for a German corpus.
  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
-  # embedding model invalidates every stored vector, so a swap means the LLM
-  # index has to be rebuilt.
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1
@@ -1,34 +0,0 @@
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: paperless-eval
-  namespace: flux-system
-spec:
-  # A throwaway copy of paperless for measuring the local LLM against the real
-  # corpus. Its Cluster restores from paperless's backup store and archives
-  # nothing back; see kubernetes/apps/paperless-eval/cluster.yaml.
-  #
-  # prune: true and no prune-disabled annotations anywhere underneath — the
-  # data here is a copy, and being able to delete the whole namespace by
-  # removing this file is the point.
-  interval: 10m
-  retryInterval: 1m
-  # 30m is not enough: the first start restores the database and then builds
-  # the search index over ~3900 documents before the webserver answers.
-  timeout: 60m
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./kubernetes/apps/paperless-eval
-  prune: true
-  wait: true
-  dependsOn:
-    - name: infrastructure
-    - name: infrastructure-crds
-    - name: infrastructure-ceph-cluster
-    - name: infrastructure-cnpg-barman
-  healthCheckExprs:
-    - apiVersion: postgresql.cnpg.io/v1
-      kind: Cluster
-      failed: "has(status.phase) && status.phase.startsWith('Failed')"
-      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"
@@ -65,21 +65,6 @@ spec:
          - 45.157.177.26/32
          - 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
          - 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
-          # netbird.renner.uno — management, signal, relay and TURN.
-          #
-          # Not optional once a bouncer is armed. Its chain sits on the INPUT
-          # hook, the proxy hosts run no other firewall, and there is therefore
-          # no conntrack rule accepting established flows ahead of it. A ban on
-          # this address would drop the RETURN traffic of the proxies' own
-          # outbound control connection and take them off the mesh — with no
-          # inbound attack involved at all.
-          #
-          # Checked when this was added: none of these addresses appear in the
-          # current decision set. That is the state today, not a guarantee —
-          # these are rented addresses on a hoster whose ranges do end up on
-          # community blocklists.
-          - 162.55.50.97/32
-          - 2a01:4f8:c2c:4579::1/128

    # mTLS between agents and the LAPI. Do not switch back to password auth:
    # the LAPI runs without persistence, so its machine-registration DB is
@@ -123,24 +108,6 @@ spec:

    lapi:
      replicas: 1
-      service:
-        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
-        #
-        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
-        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
-        # annotations, and on an Ingress they are silently ignored. The symptom
-        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
-        # answers with "Client sent an HTTP request to an HTTPS server." and a
-        # 400 — an error that looks like a malformed request rather than a
-        # scheme mismatch.
-        #
-        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
-        # plaintext port, so https here is mandatory rather than a hardening
-        # choice. The ServersTransport carries the chart's own CA, since that
-        # certificate is self-signed.
-        annotations:
-          traefik.ingress.kubernetes.io/service.serversscheme: https
-          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
      # Far below the chart default, which reserves half a core per LAPI for a
      # workload that draws single-digit millicores — enough to leave the
      # cluster unschedulable with one node out. Only the reservation shrinks;
@@ -217,32 +184,6 @@ spec:
        - namespace: ingress
          podName: traefik-*
          program: traefik
-          # Read by polling, not inotify. Every path under
-          # /var/log/containers is a symlink into /var/log/pods, and kubelet
-          # rotates the target once it reaches containerLogMaxSize (10Mi,
-          # roughly every eight hours for Traefik here). An inotify tailer
-          # does not survive that rotation: it reopens the path, waits for a
-          # file that is in fact already present, and never reads another
-          # line. Nothing about that state looks broken from outside — the
-          # pod stays Running and Ready, the LAPI keeps answering, and the
-          # cumulative counters keep displaying whatever they reached, so the
-          # only symptom is that no scenario ever fires again.
-          #
-          # CrowdSec detects the hazard itself and logs it at startup, with
-          # this exact remedy in the message: "is a symlink, but inotify
-          # polling is enabled. Crowdsec will not be able to detect rotation.
-          # Consider setting poll_without_inotify to true". Upstream carries
-          # the same case as crowdsec#2449 (k3s + Traefik), and the chart
-          # gained this field in helm-charts#109 specifically to fix it.
-          #
-          # Orthogonal to the force_inotify: true that the chart hardcodes
-          # into acquis.yaml and offers no way to unset. That one governs the
-          # fsnotify watch on the directory, i.e. discovering the log of a
-          # replacement Traefik pod; this one governs how an already
-          # discovered file is read. Both true is the supported combination.
-          #
-          # Costs CPU per open file per upstream docs — two files per node.
-          poll_without_inotify: true
      env:
        - name: COLLECTIONS
          value: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules"
@@ -44,8 +44,8 @@ metadata:
  namespace: crowdsec
 spec:
  serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAs:
-    - secret: crowdsec-root-secret
+  rootCAsSecrets:
+    - crowdsec-root-secret
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
@@ -60,12 +60,8 @@ metadata:
    cert-manager.io/cluster-issuer: letsencrypt-cluster
    # NOT behind the Authentik gate: the client is a daemon with an API key,
    # not a browser that can complete an OIDC flow.
-    #
-    # The backend scheme and ServersTransport are NOT set here. Under the
-    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
-    # are SERVICE annotations — on an Ingress they are silently ignored, and
-    # Traefik then talks plaintext to a TLS-only backend. They live on the
-    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
+    traefik.ingress.kubernetes.io/service.serversscheme: https
+    traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
 spec:
  ingressClassName: traefik
  rules:
@@ -69,13 +69,6 @@ kind: Namespace
 metadata:
  name: paperless
 ---
-# A disposable copy of paperless's database, for measuring how the local LLM
-# classifies documents without experimenting on the live archive.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: paperless-eval
---
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -34,33 +34,6 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # The paperless-eval namespace reaches the embedding endpoint for the same
-    # reason paperless does; its chat model runs inside each eval job's own pod
-    # and needs no grant here.
-    #
-    # Scoped to the namespace rather than to an app label, unlike the grants
-    # around it: that namespace exists only to run evaluations, and its
-    # workload is jobs created per measurement whose labels vary with what is
-    # being measured. A label match would be a trap that presents as an
-    # embedding call silently dropped.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
-    # Home Assistant's llama.cpp conversation agent calls the chat model only —
-    # it has no use for the embedding endpoint, so unlike paperless above this
-    # grant does not extend to llama-embed.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": home-assistant
-            app.kubernetes.io/name: home-assistant
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": monitoring
@@ -167,18 +167,6 @@ spec:
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": influxdb
-    # The cluster's own chat model, for the llama.cpp conversation agent that
-    # backs Assist. Scoped to the one pod and port rather than the ai namespace
-    # as a whole, so this grant cannot widen if something else is placed there —
-    # the embedding endpoint next to it is deliberately not included.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-server
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
    # traefik for east-west OIDC roundtrips that go through the
    # public hostname via k8s_gateway.
    - toEndpoints:
@@ -6,7 +6,6 @@ resources:
  - ingress-isolation.yaml
  - authentik-ns.yaml
  - paperless-ns.yaml
-  - paperless-eval-ns.yaml
  - audiobookshelf-ns.yaml
  - home-assistant-ns.yaml
  - spegel-ns.yaml
@@ -1,141 +0,0 @@
-# Narrower than paperless-ns.yaml, because this namespace serves nobody.
-#
-# There is no ingress, no gate outpost and no hostname: the only things that
-# talk to it are its own pods. Monitoring is admitted so a run can be watched,
-# and host/remote-node for the kubelet probes.
-#
-# Egress reaches the ai namespace for the embedding endpoint, which is the one
-# dependency an eval job has outside its own pod, and world:443 so the CNPG
-# recovery can read the backup store on Hetzner.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: default-deny-ingress
-  namespace: paperless-eval
-spec:
-  endpointSelector: {}
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": monitoring
-    - fromEntities:
-        - host
-        - remote-node
-    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
-    # home-assistant-ns.yaml for the rationale.
-    - fromEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
-  egress:
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    # The embedding endpoint. The chat model runs inside each job's own pod, so
-    # it needs no grant; the embedder is shared with production because it is
-    # read-only and cheap, and swapping it is a phase-two concern.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-embed
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": rook-ceph
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": kube-system
-            k8s-app: kube-dns
-      toPorts:
-        - ports:
-            - port: "53"
-              protocol: UDP
-    - toEntities:
-        - host
-        - remote-node
-    # world:443 — the CNPG recovery reads the backup store on Hetzner Object
-    # Storage, and image pulls.
-    - toEntities:
-        - world
-      toPorts:
-        - ports:
-            - port: "443"
-              protocol: TCP
-    # ICMP — egress mirror; see ingress block above.
-    - toEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
---
-# The CNPG operator polls each instance's status endpoint over HTTPS
-# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
-# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
-# while Postgres serves normally — issue #433, where the operator's dial timed
-# out because the apps baseline never allowed the database namespace.
-#
-# It matters here even for a throwaway: the Flux Kustomization waits on
-# Cluster status, and a smoke test that cannot report its own health is not one.
-#
-# Scoped to the pprleval-pg endpoints and that one port, like the equivalent
-# rules for wp-pg and ha-pg: the barman and mariadb operators sharing the
-# database namespace have no reason to reach anything here.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: pprleval-pg-allow-cnpg-operator
-  namespace: paperless-eval
-spec:
-  endpointSelector:
-    matchLabels:
-      cnpg.io/cluster: pprleval-pg
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": database
-            app.kubernetes.io/name: cloudnative-pg
-      toPorts:
-        - ports:
-            - port: "8000"
-              protocol: TCP
@@ -27,7 +27,6 @@ spec:
        - ingress
        - monitoring
        - paperless
-        - paperless-eval # same apps/paperless key plus the shared barman S3 creds
        - trmnl
        - velero
        - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)
@@ -0,0 +1,207 @@
+#!/usr/bin/env bash
+# End-to-end evaluation of Paperless document suggestions against the real
+# archive, comparing the two engines Paperless ships:
+#
+#   /api/documents/<id>/suggestions/      the built-in scikit-learn classifier,
+#                                         trained on the existing corpus
+#   /api/documents/<id>/ai_suggestions/   the LLM path, optionally with RAG
+#
+# WHY THIS EXISTS
+#
+# scripts/embed-eval-real.sh measures whether a document's nearest neighbours
+# carry its labels. That is a proxy: Paperless does not act on the neighbours
+# directly, it hands them to an LLM as context, alongside the document text and
+# a candidate list of existing tags and correspondents. The LLM can override
+# poor context, or fail with good context. So a better embedding score does not
+# automatically mean better suggestions, and nothing so far has verified that it
+# does.
+#
+# This measures the thing that is actually wanted: does Paperless suggest the
+# labels a human already chose? Everything else — MTEB scores, retrieval
+# accuracy — is only interesting insofar as it predicts this number.
+#
+# It also answers a question that predates the LLM work: with ten years of
+# hand-curated labels, the built-in classifier has excellent training data. It
+# cannot invent a new tag or a title, which is the LLM's advantage — but for
+# assigning labels that already exist it may simply be better. Running both on
+# the same documents settles that rather than assuming it.
+#
+# Ground truth is the current filing. Where the two disagree with it, that is
+# not automatically an error: over ten years filing conventions change, and an
+# old document may carry an older convention. Treat the absolute numbers as
+# indicative and the comparison between engines as the result.
+#
+# Requires a Paperless API token in PAPERLESS_TOKEN. The API is only reachable
+# from inside the cluster (the namespace policy admits the Authentik gate
+# outpost alone), so requests are issued from within the paperless pod.
+#
+# Usage:
+#   PAPERLESS_TOKEN=... scripts/paperless-eval.sh [-n 100] [--engine both|classic|ai]
+set -uo pipefail
+
+SAMPLE="${PAPERLESS_EVAL_SAMPLE:-100}"
+ENGINE=both
+PG_POD="${PAPERLESS_EVAL_PG_POD:-pprl-pg-2}"
+APP_POD="${PAPERLESS_EVAL_APP_POD:-deploy/paperless}"
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -n|--sample) SAMPLE="$2"; shift 2 ;;
+        --engine)    ENGINE="$2"; shift 2 ;;
+        -h|--help)   sed -n '2,40p' "$0"; exit 0 ;;
+        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
+    esac
+done
+
+if [ -z "${PAPERLESS_TOKEN:-}" ]; then
+    echo "PAPERLESS_TOKEN is not set." >&2
+    echo "Create one for an existing user, then remove it when finished." >&2
+    exit 2
+fi
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+# Deterministic sample, so repeated runs and different engines see the same
+# documents. Only documents that carry both labels are useful as ground truth.
+echo "reading $SAMPLE labelled documents from postgres ($PG_POD)..."
+# Names come along with the ids because the AI engine reports an unresolved
+# guess as a name string rather than an id; see the lenient scoring below.
+# Tab-separated rather than '|', which occurs in correspondent names.
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tA -F$'\t' -c \
+  "select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
+   from documents_document d
+   join documents_correspondent c on c.id = d.correspondent_id
+   join documents_documenttype  t on t.id = d.document_type_id
+   where d.correspondent_id is not null
+     and d.document_type_id is not null
+     and length(d.content) > 400
+   order by md5(d.id::text)
+   limit $SAMPLE;" > "$WORK/docs.txt" 2>"$WORK/pg-err.txt"
+if ! [ -s "$WORK/docs.txt" ]; then
+    echo "no documents returned from postgres." >&2
+    [ -s "$WORK/pg-err.txt" ] && cat "$WORK/pg-err.txt" >&2
+    exit 2
+fi
+N=$(grep -c . "$WORK/docs.txt")
+
+# The floor: always guessing the most frequent label. Without it a number like
+# "70% correct" cannot be read, because a skewed archive gives that away for
+# free — one correspondent holds several hundred of ~3900 documents.
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
+  "select max(c) from (select count(*) c from documents_document
+   where correspondent_id is not null group by correspondent_id) x;" \
+  > "$WORK/corr_max.txt" 2>"$WORK/pg-err.txt"
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
+  "select count(*) from documents_document where correspondent_id is not null;" \
+  > "$WORK/corr_total.txt" 2>"$WORK/pg-err.txt"
+
+api() { # path -> body on stdout
+    kubectl -n paperless exec "$APP_POD" -c paperless -- \
+        curl -sS -m 300 -H "Authorization: Token $PAPERLESS_TOKEN" \
+        "http://127.0.0.1:8000$1" 2>"$WORK/err.txt"
+}
+
+# A hit under two definitions, because the AI engine can be right in a way the
+# API response does not record as a suggestion:
+#
+#   strict   the true label's id appears in `.correspondents` /
+#            `.document_types`. This is what the UI actually offers, so it is
+#            the number that describes the feature as experienced.
+#
+#   lenient  strict, OR the true label's name is contained wordwise in one of
+#            `.suggested_correspondents` / `.suggested_document_types`.
+#
+# The gap between them is one specific failure: the LLM names the right entity
+# but paperless_ai.matching cannot resolve the name to the existing object,
+# because difflib.get_close_matches runs at MATCH_THRESHOLD = 0.8 and a correct
+# but more verbose answer falls below it — observed with "Generali Deutschland
+# Krankenversicherung AG" against the existing "Generali AG", which yielded an
+# empty `correspondents` list.
+#
+# Reporting only the strict number would blame the model for a matcher problem;
+# reporting only the lenient one would claim a usefulness the UI does not
+# deliver. The classic engine emits no `suggested_*` fields at all, so both
+# numbers coincide for it and the comparison stays fair in either column.
+LENIENT_JQ='
+def norm: ascii_downcase | gsub("[^\\p{L}\\p{N}]+"; " ") | split(" ") | map(select(length > 0));
+def contains_all($want): . as $have | $want | all(. as $t | $have | index([$t]) != null);
+($want_name | norm) as $w
+| ((.[$id_field] // []) | index($want_id) != null)
+  or ([(.[$name_field] // [])[] | norm] | any(contains_all($w)))
+'
+
+# score <engine> <url-suffix>
+score() {
+    local engine="$1" suffix="$2"
+    local n=0 corr_hit=0 type_hit=0 corr_len=0 type_len=0 fails=0 start elapsed
+    echo ""
+    echo "=== $engine"
+    start=$(date +%s)
+    while IFS=$'\t' read -r id want_corr want_type want_corr_name want_type_name; do
+        [ -n "$id" ] || continue
+        n=$((n + 1))
+        body=$(api "/api/documents/$id/$suffix")
+        if ! printf '%s' "$body" | jq -e . >/dev/null 2>&1; then
+            fails=$((fails + 1))
+            [ "$fails" -le 3 ] && printf '    %s: keine gueltige Antwort: %s\n' \
+                "$id" "$(printf '%s' "$body" | tr '\n' ' ' | head -c 120)" >&2
+            continue
+        fi
+        if printf '%s' "$body" | jq -e --argjson w "$want_corr" \
+               '.correspondents // [] | index($w) != null' >/dev/null 2>&1; then
+            corr_hit=$((corr_hit + 1))
+        fi
+        if printf '%s' "$body" | jq -e --argjson w "$want_type" \
+               '.document_types // [] | index($w) != null' >/dev/null 2>&1; then
+            type_hit=$((type_hit + 1))
+        fi
+        if printf '%s' "$body" | jq -e \
+               --argjson want_id "$want_corr" --arg want_name "$want_corr_name" \
+               --arg id_field correspondents --arg name_field suggested_correspondents \
+               "$LENIENT_JQ" >/dev/null 2>&1; then
+            corr_len=$((corr_len + 1))
+        fi
+        if printf '%s' "$body" | jq -e \
+               --argjson want_id "$want_type" --arg want_name "$want_type_name" \
+               --arg id_field document_types --arg name_field suggested_document_types \
+               "$LENIENT_JQ" >/dev/null 2>&1; then
+            type_len=$((type_len + 1))
+        fi
+        [ $((n % 25)) -eq 0 ] && echo "    $n/$N"
+    done < "$WORK/docs.txt"
+    elapsed=$(( $(date +%s) - start ))
+    pct() { awk -v a="$1" -v b="$2" 'BEGIN{if (b>0) printf "%.1f", 100*a/b; else print "0.0"}'; }
+    printf '    correspondent, offered  : %5s%%   recognised: %5s%%\n' \
+        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")"
+    printf '    document type, offered  : %5s%%   recognised: %5s%%\n' \
+        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")"
+    printf '    no usable answer        : %s of %s\n' "$fails" "$n"
+    printf '    %ss total, %.1fs per document\n' "$elapsed" "$(awk -v e="$elapsed" -v n="$n" 'BEGIN{print e/n}')"
+    printf '%-9s %7s %7s %7s %7s %6s %8ss\n' "$engine" \
+        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")" \
+        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")" \
+        "$fails" "$elapsed" >> "$WORK/summary.txt"
+}
+
+: > "$WORK/summary.txt"
+FLOOR=$(awk -v m="$(cat "$WORK/corr_max.txt")" -v t="$(cat "$WORK/corr_total.txt")" \
+    'BEGIN{if (t>0) printf "%.1f", 100*m/t; else print "?"}')
+echo "documents: $N   most-frequent-correspondent floor: ${FLOOR}%"
+
+case "$ENGINE" in
+    both)    score classic suggestions/; score ai ai_suggestions/ ;;
+    classic) score classic suggestions/ ;;
+    ai)      score ai ai_suggestions/ ;;
+    *) echo "unknown engine: $ENGINE" >&2; exit 2 ;;
+esac
+
+echo ""
+echo "================================================================"
+printf '%-9s %7s %7s %7s %7s %6s %9s\n' \
+    ENGINE CORR CORR+ TYPE TYPE+ FAILS TIME
+cat "$WORK/summary.txt"
+printf '%-9s %7s\n' "floor" "$FLOOR"
+echo "================================================================"
+echo "CORR/TYPE  = true label offered as a resolved id (what the UI shows)"
+echo "CORR+/TYPE+ = also counting names the LLM got right but matching dropped"
@@ -1,72 +0,0 @@
-# Measuring Paperless suggestions against the real corpus
-
-Compares the two engines Paperless ships — the scikit-learn classifier and the
-LLM path — over documents whose labels a human already chose, in the
-`paperless-eval` namespace rather than against the live archive.
-
-**Applied by hand, not by Flux.** A Job is immutable, so re-running a
-configuration means deleting and recreating it, and a Flux-managed Job would
-simply come back. The namespace itself *is* managed
-(`kubernetes/apps/paperless-eval/`); only the measurement is not.
-
-## Running one
-
-```sh
-kubectl -n paperless-eval create configmap paperless-eval-script \
-  --from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
-  | kubectl apply -f -
-
-kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
-kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
-```
-
-Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
-document) and `<label>.summary.json`, and in the pod log.
-
-Re-running after editing `eval.py` means re-creating the ConfigMap and deleting
-the Job first — `kubectl -n paperless-eval delete job eval-qwen35`.
-
-## Adding a candidate
-
-Copy `job-qwen35.yaml`, then change four things: the Job name, the
-`--hf-repo`/`--alias` on the `llama-server` sidecar, `PAPERLESS_AI_LLM_MODEL` on
-the `paperless` sidecar (it must match the alias), and `EVAL_LABEL` on the
-`eval` container (it names the output files).
-
-Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1
-and k8s-2 have a free iGPU. A third waits.
-
-## Reading the output
-
-`complement` in the summary is the number the exercise exists for. The two
-engines are not alternatives: the classifier costs 0.6 s per document and keeps
-running whatever the LLM does, so what matters is what the LLM adds where the
-classifier is wrong — `llm_rescued` against `llm_only_wrong`. Two aggregate
-percentages cannot show that; 74% against 72% could overlap almost entirely or
-barely at all.
-
-`correspondent_offered` against `correspondent_recognised` is the
-`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
-rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
-stored "Generali AG".
-
-`type_specific` against `type_catchall` splits on whether the true type is
-"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
-semantic content. The classifier can memorise it; a model reading the document
-has to infer "none of the above".
-
-## What to watch for
-
-Two things make a result look like a bad model when it is not:
-
-**Prompt length.** Past roughly 14k tokens this stack stops producing language
-and starts producing garbage or nothing, without any error — and a single such
-request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
-near 5k, but the 252 documents over 14000 characters push toward the boundary
-with their RAG neighbours. A run that starts scoring zeros mid-way is more
-likely this than the model.
-
-**Ground truth.** It is the current filing, and over ten years conventions
-change. A disagreement with an old document is not automatically an error. The
-comparison between engines is the durable result; the absolute numbers are
-indicative.
@@ -1,390 +0,0 @@
-#!/usr/bin/env python3
-"""Measure what Paperless suggests for documents whose labels a human already chose.
-
-Runs both engines against the same documents:
-
-    /api/documents/<id>/suggestions/      the built-in scikit-learn classifier
-    /api/documents/<id>/ai_suggestions/   the LLM path, with RAG
-
-WHAT THIS ANSWERS, AND WHAT AN EARLIER VERSION DID NOT
-
-The two engines are not alternatives — the classifier costs 0.6 s per document
-and keeps running whatever the LLM does. So the useful question is not "which
-scores higher" but "what does the LLM add where the classifier is wrong", and
-that cannot be recovered from two aggregate percentages: 74% and 72% could
-overlap almost entirely or barely at all, and the difference between those two
-worlds is the entire value of running an LLM.
-
-So every document is written out individually, with both engines' verdicts, and
-the summary is derived from that rather than counted along the way.
-
-SCORING
-
-Correspondent and document type are each scored twice:
-
-    offered      the true label's id came back in `correspondents` /
-                 `document_types` — what the UI actually presents
-    recognised   also counting the true label's name appearing wordwise in
-                 `suggested_correspondents` / `suggested_document_types`
-
-The gap between them is one specific failure: the model names the right entity
-but paperless_ai.matching cannot resolve it, because difflib runs at
-MATCH_THRESHOLD = 0.8 and a correct but more verbose answer falls below it —
-"Generali Deutschland Krankenversicherung AG" against the existing "Generali
-AG". Reporting only the first blames the model for a matcher problem; only the
-second claims a usefulness the UI does not deliver.
-
-Document type is additionally split on whether the true type is "Sonstige",
-which holds 1133 of ~3900 documents and is the one label with no semantic
-content — a restaurant reservation lands there because nothing else fits. The
-classifier can memorise that; a model reading the document has to infer "none
-of the above". Scored separately rather than dropped, because the model does
-sometimes get it right.
-
-GROUND TRUTH
-
-The current filing, which deserves care: over ten years conventions change, and
-a disagreement with an old document is not automatically an error. Treat the
-comparison between engines as the result and the absolute numbers as
-indicative.
-
-Env:
-    PAPERLESS_URL     default http://127.0.0.1:8000
-    PGHOST/PGDATABASE standard libpq, for ground truth and the API token
-    EVAL_SAMPLE       documents to score, default 50
-    EVAL_ENGINES      comma-separated subset of classic,ai — default both
-    EVAL_LABEL        free-text name for this configuration, recorded in the
-                      output so results from different jobs stay distinguishable
-    EVAL_OUT          directory for results, default /results
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import os
-import re
-import secrets
-import sys
-import time
-import urllib.error
-import urllib.request
-
-import psycopg
-
-PAPERLESS_URL = os.environ.get("PAPERLESS_URL", "http://127.0.0.1:8000")
-SAMPLE = int(os.environ.get("EVAL_SAMPLE", "50"))
-ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
-LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
-OUT_DIR = os.environ.get("EVAL_OUT", "/results")
-
-# A backstop, not the real bound. Generation is capped server-side by
-# --n-predict on the llama-server sidecar, which is what actually stops a model
-# that will not terminate; this catches everything else — a wedged sidecar, a
-# request that never reaches it.
-#
-# 300 s against the ~73 s a healthy document takes end to end, prompt included.
-# It does work: llama-server logs `stop: cancel task` on the disconnect and
-# starts the next request 0.4 s later.
-REQUEST_TIMEOUT = 300
-
-CATCHALL_TYPE = "Sonstige"
-
-
-def log(msg: str) -> None:
-    print(f"[eval] {msg}", flush=True)
-
-
-def api_token(conn) -> str:
-    """A token for this run, created in the database.
-
-    The API needs one and this database is a disposable clone, so the run
-    issues its own against a superuser rather than depending on a token that
-    happened to come across in the restore.
-    """
-    token = secrets.token_hex(20)
-    with conn.cursor() as cur:
-        cur.execute(
-            "select id from auth_user where is_superuser and is_active order by id limit 1",
-        )
-        row = cur.fetchone()
-        if row is None:
-            raise SystemExit("no active superuser in the restored database")
-        user_id = row[0]
-        cur.execute("delete from authtoken_token where user_id = %s", (user_id,))
-        cur.execute(
-            "insert into authtoken_token (key, created, user_id) values (%s, now(), %s)",
-            (token, user_id),
-        )
-    conn.commit()
-    return token
-
-
-def sample(conn) -> list[dict]:
-    """The documents to score, ordered by md5 of the id.
-
-    Deterministic, so every engine and every configuration sees the same
-    documents and results stay comparable across jobs. Only documents carrying
-    both labels are usable as ground truth.
-    """
-    with conn.cursor() as cur:
-        cur.execute(
-            """
-            select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
-              from documents_document d
-              join documents_correspondent c on c.id = d.correspondent_id
-              join documents_documenttype  t on t.id = d.document_type_id
-             where d.correspondent_id is not null
-               and d.document_type_id is not null
-               and length(d.content) > 400
-             order by md5(d.id::text)
-             limit %s
-            """,
-            (SAMPLE,),
-        )
-        return [
-            {
-                "id": r[0],
-                "correspondent_id": r[1],
-                "document_type_id": r[2],
-                "correspondent_name": r[3],
-                "document_type_name": r[4],
-            }
-            for r in cur.fetchall()
-        ]
-
-
-def floor(conn) -> float:
-    """Always guessing the most frequent correspondent.
-
-    Without it a number like "70% correct" cannot be read: one correspondent
-    holds several hundred of ~3900 documents.
-    """
-    with conn.cursor() as cur:
-        cur.execute(
-            """
-            select max(c), sum(c) from (
-                select count(*) c from documents_document
-                 where correspondent_id is not null group by correspondent_id
-            ) x
-            """,
-        )
-        top, total = cur.fetchone()
-    # float() on both: psycopg returns Postgres numerics as Decimal, and
-    # mixing that with a float literal raises TypeError rather than coercing.
-    return 100.0 * float(top) / float(total) if total else 0.0
-
-
-def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
-    req = urllib.request.Request(
-        f"{PAPERLESS_URL}{path}",
-        headers={"Authorization": f"Token {token}"},
-    )
-    start = time.monotonic()
-    try:
-        with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT) as resp:
-            body = resp.read()
-        return json.loads(body), None, time.monotonic() - start
-    except urllib.error.HTTPError as err:
-        return None, f"HTTP {err.code}: {err.read()[:200]!r}", time.monotonic() - start
-    except Exception as err:  # noqa: BLE001 — a failed document is data, not a crash
-        return None, f"{type(err).__name__}: {err}", time.monotonic() - start
-
-
-def normalise(text: str) -> set[str]:
-    return {w for w in re.split(r"[^\w]+", text.lower()) if w}
-
-
-def recognised(payload: dict, field: str, want_name: str) -> bool:
-    """True when a suggested name contains every word of the true label's name.
-
-    Handles the observed failure directly: the model answers with a superset of
-    the stored name, which difflib rejects at 0.8 but a reader would accept.
-    """
-    want = normalise(want_name)
-    return any(want <= normalise(name) for name in payload.get(field) or [])
-
-
-def degenerate(payload: dict) -> str | None:
-    """Whether this looks like broken inference rather than a wrong guess.
-
-    Past roughly 14k prompt tokens this stack stops producing language — first
-    empty completions, then runs of digits and CJK characters — and one such
-    request poisons the llama.cpp slot until the pod restarts. Nothing in the
-    HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
-
-    Scored as a wrong answer it would silently depress a candidate's numbers,
-    and because the slot stays poisoned it would take the rest of the run with
-    it. Flagged instead, so the failure is visible as itself.
-    """
-    title = payload.get("title") or ""
-    if not title.strip():
-        return "empty title"
-    # Digits, punctuation and whitespace are normal in a document title —
-    # "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
-    # earlier version of this check. What broken inference produces is
-    # characters from other scripts, so look for those directly rather than
-    # inferring them from a shortage of Latin ones.
-    foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
-    if foreign > 0.2 * len(title):
-        return f"non-latin title: {title[:40]!r}"
-    return None
-
-
-def score_document(payload: dict, doc: dict) -> dict:
-    offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
-    offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
-    return {
-        "correspondent_offered": offered_corr,
-        "correspondent_recognised": offered_corr
-        or recognised(payload, "suggested_correspondents", doc["correspondent_name"]),
-        "type_offered": offered_type,
-        "type_recognised": offered_type
-        or recognised(payload, "suggested_document_types", doc["document_type_name"]),
-        "suggested_correspondents": payload.get("suggested_correspondents") or [],
-        "suggested_document_types": payload.get("suggested_document_types") or [],
-        "title": payload.get("title"),
-    }
-
-
-def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
-    log(f"=== {engine}: {len(docs)} documents")
-    rows = []
-    for n, doc in enumerate(docs, 1):
-        payload, error, elapsed = fetch(
-            f"/api/documents/{doc['id']}/{suffix}",
-            token,
-        )
-        row = {
-            "engine": engine,
-            "document_id": doc["id"],
-            "seconds": round(elapsed, 1),
-            "truth_correspondent": doc["correspondent_name"],
-            "truth_type": doc["document_type_name"],
-            "catchall_type": doc["document_type_name"] == CATCHALL_TYPE,
-        }
-        if payload is None:
-            row["error"] = error
-            log(f"  {doc['id']}: {error}")
-        else:
-            row.update(score_document(payload, doc))
-            # Only the AI path returns a title; the classifier answers with ids
-            # alone, so an "empty title" there is normal rather than broken.
-            if engine == "ai" and (broken := degenerate(payload)) is not None:
-                row["degenerate"] = broken
-                log(f"  {doc['id']}: DEGENERATE — {broken}")
-        rows.append(row)
-        if n % 10 == 0:
-            log(f"  {n}/{len(docs)}")
-    return rows
-
-
-def pct(hits: int, total: int) -> float:
-    return round(100.0 * hits / total, 1) if total else 0.0
-
-
-def summarise(rows: list[dict], engine: str) -> dict:
-    mine = [r for r in rows if r["engine"] == engine]
-    # Degenerate rows are excluded from the accuracy figures: they measure the
-    # backend giving up, not the model choosing wrongly, and leaving them in
-    # would quietly punish whichever candidate happened to trip the slot.
-    scored = [r for r in mine if "error" not in r and "degenerate" not in r]
-    failed = [r for r in mine if "error" in r]
-    broken = [r for r in mine if "degenerate" in r]
-    specific = [r for r in scored if not r["catchall_type"]]
-    catchall = [r for r in scored if r["catchall_type"]]
-    return {
-        "engine": engine,
-        "scored": len(scored),
-        "failed": len(failed),
-        # Non-zero means the run is suspect from that point on, not that the
-        # model is bad — the slot stays poisoned until the pod restarts.
-        "degenerate": len(broken),
-        "seconds_total": round(sum(r["seconds"] for r in mine)),
-        "correspondent_offered": pct(
-            sum(r["correspondent_offered"] for r in scored), len(scored)
-        ),
-        "correspondent_recognised": pct(
-            sum(r["correspondent_recognised"] for r in scored), len(scored)
-        ),
-        "type_offered": pct(sum(r["type_offered"] for r in scored), len(scored)),
-        "type_specific": pct(
-            sum(r["type_offered"] for r in specific), len(specific)
-        ),
-        "type_catchall": pct(
-            sum(r["type_offered"] for r in catchall), len(catchall)
-        ),
-    }
-
-
-def complement(rows: list[dict]) -> dict | None:
-    """What the LLM adds where the classifier is wrong, and what it costs.
-
-    The reason this script writes per-document rows at all. Only computed when
-    both engines ran, and only over documents both actually scored.
-    """
-    by_engine = {"classic": {}, "ai": {}}
-    for row in rows:
-        usable = "error" not in row and "degenerate" not in row
-        if usable and row["engine"] in by_engine:
-            by_engine[row["engine"]][row["document_id"]] = row
-    both = set(by_engine["classic"]) & set(by_engine["ai"])
-    if not both or not by_engine["ai"]:
-        return None
-
-    rescued = missed = broken = agreed = 0
-    for doc_id in both:
-        classic_hit = by_engine["classic"][doc_id]["correspondent_offered"]
-        ai_hit = by_engine["ai"][doc_id]["correspondent_offered"]
-        if classic_hit and ai_hit:
-            agreed += 1
-        elif not classic_hit and ai_hit:
-            rescued += 1
-        elif classic_hit and not ai_hit:
-            broken += 1
-        else:
-            missed += 1
-    return {
-        "documents": len(both),
-        "both_right": agreed,
-        "llm_rescued": rescued,
-        "llm_only_wrong": broken,
-        "both_wrong": missed,
-    }
-
-
-def main() -> int:
-    os.makedirs(OUT_DIR, exist_ok=True)
-    with psycopg.connect() as conn:
-        token = api_token(conn)
-        docs = sample(conn)
-        guess_floor = floor(conn)
-    log(f"label={LABEL} documents={len(docs)} floor={guess_floor:.1f}%")
-
-    rows: list[dict] = []
-    for engine, suffix in (("classic", "suggestions/"), ("ai", "ai_suggestions/")):
-        if engine in ENGINES:
-            rows.extend(run(engine, suffix, docs, token))
-
-    result = {
-        "label": LABEL,
-        "sample": len(docs),
-        "floor": round(guess_floor, 1),
-        "summaries": [summarise(rows, e) for e in ENGINES if e in ("classic", "ai")],
-        "complement": complement(rows),
-    }
-
-    with open(f"{OUT_DIR}/{LABEL}.jsonl", "w") as handle:
-        for row in rows:
-            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
-    with open(f"{OUT_DIR}/{LABEL}.summary.json", "w") as handle:
-        json.dump(result, handle, ensure_ascii=False, indent=2)
-
-    # Also to stdout, so the numbers are in the pod log and reachable via Loki
-    # without mounting the volume.
-    log(json.dumps(result, ensure_ascii=False, indent=2))
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())
@@ -1,324 +0,0 @@
---
-# One configuration under test, complete in one pod.
-#
-# A paperless instance can only point at one chat endpoint, so measuring two
-# candidates at once means two paperless instances — which makes the pod, not
-# the namespace, the right unit. Everything a measurement needs is here except
-# the database and the embedding endpoint, both of which are read-only and
-# shared.
-#
-# The three services are native sidecars (initContainers with
-# restartPolicy: Always), not regular containers. That matters twice: they
-# start in order and each must pass its probes before the next begins, which
-# is what keeps paperless from racing the database; and when the eval
-# container exits they are torn down, so the Job actually completes instead of
-# running forever behind a llama-server that never returns.
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: eval-qwen35
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: paperless-eval
-    app.kubernetes.io/component: eval
-spec:
-  # A failed run is re-read, not retried blind: a retry would spend another
-  # hour reproducing whatever went wrong.
-  backoffLimit: 0
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless-eval
-        app.kubernetes.io/component: eval
-    spec:
-      restartPolicy: Never
-      enableServiceLinks: false
-      # One measurement per node. Two jobs sharing a node would share its iGPU,
-      # and every timing in the result would then describe contention rather
-      # than the model — which is exactly how a decode rate of 4.7 tokens/s got
-      # attributed to context length once already.
-      affinity:
-        podAntiAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-            - labelSelector:
-                matchLabels:
-                  app.kubernetes.io/component: eval
-              topologyKey: kubernetes.io/hostname
-      securityContext:
-        fsGroup: 1000
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      initContainers:
-        # A private copy of /data — the search index and, the expensive part,
-        # the LLM index the RAG path needs.
-        #
-        # It cannot be mounted live and shared: LLM_INDEX_DIR is hardcoded to
-        # DATA_DIR / "llm_index", the store behind it is SQLite, and SQLite
-        # locking across pods over CephFS is not dependable. Nor can it be left
-        # out — an empty llm_index makes retrieve_similar_nodes return nothing,
-        # and the job would quietly measure the no-RAG path while reporting a
-        # RAG result.
-        #
-        # ~160 MiB, so seconds. A regular initContainer: it must finish before
-        # the sidecars start.
-        - name: seed-data
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          command:
-            - sh
-            - -c
-            - |
-              set -e
-              # -r, not -a: preserving ownership needs CHOWN, which this
-              # container does not have, and cp then exits non-zero after
-              # copying the files anyway. Running as 1000 makes the copy land
-              # with the ownership paperless needs — it writes to the LLM index
-              # while serving.
-              cp -r /golden/. /data/
-              du -sh /data/index /data/llm_index
-              # Refuse to start rather than measure the wrong thing. An empty
-              # llm_index makes retrieve_similar_nodes return nothing, and the
-              # run would report a RAG result having never retrieved anything —
-              # which is exactly what the first attempt at this job did, because
-              # the golden volume had not been populated yet. The index is
-              # ~125 MiB once built; anything under 1 MiB is not one.
-              bytes=$(du -sb /data/llm_index | cut -f1)
-              if [ "$bytes" -lt 1000000 ]; then
-                echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
-                echo "smoke-test deployment first:" >&2
-                echo "  kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
-                echo "    python manage.py document_llmindex rebuild" >&2
-                exit 1
-              fi
-          volumeMounts:
-            - name: golden
-              mountPath: /golden
-              readOnly: true
-            - name: data
-              mountPath: /data
-          securityContext:
-            runAsUser: 1000
-            runAsGroup: 1000
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: ["ALL"]
-        - name: llama-server
-          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
-          restartPolicy: Always
-          args:
-            - --hf-repo
-            - ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
-            - --alias
-            - qwen3.6-35b-a3b
-            - --n-gpu-layers
-            - "99"
-            - --threads
-            - "4"
-            # The production ceiling. Not because the flag value itself breaks
-            # anything — bisection showed the corruption boundary is prompt
-            # length, around 14000 tokens, whatever --ctx-size says — but
-            # because measuring against a different ceiling than production
-            # runs would not describe production.
-            - --ctx-size
-            - "16384"
-            - --parallel
-            - "1"
-            - --ubatch-size
-            - "1024"
-            # The default is -1, infinity: nothing on the server caps how long
-            # a request generates, and paperless sets no budget of its own and
-            # waits. A classification that needs ~300 tokens ran to 10269 at a
-            # healthy 11 tok/s, and three of the first eleven documents did it
-            # — 900 s each, most of an hour spent on four documents.
-            #
-            # 768 is generous against the ~300 a correct answer takes, and a
-            # truncated response is better evidence than a timeout: it shows
-            # what the model was doing instead of stopping.
-            - --n-predict
-            - "768"
-            # Without this, Qwen3.6 spends the response on reasoning_content
-            # and returns empty content — 983 tokens into a classification that
-            # needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
-            # slow or broken model and is neither.
-            #
-            # A candidate whose template has no such switch has to be measured
-            # as it comes, and that is a fair part of its result: paperless
-            # gives the model no token budget and waits.
-            - --chat-template-kwargs
-            - '{"enable_thinking": false}'
-            - --no-webui
-          env:
-            - name: LLAMA_CACHE
-              value: /models
-          ports:
-            - name: llm
-              containerPort: 8080
-          volumeMounts:
-            - name: models
-              mountPath: /models
-          # ~20 GB of weights off CephFS, so a generous budget: 60 * 20s.
-          startupProbe:
-            httpGet:
-              path: /health
-              port: llm
-            periodSeconds: 20
-            failureThreshold: 60
-          resources:
-            requests:
-              cpu: "1"
-              memory: 24Gi
-              gpu.intel.com/i915: "1"
-            limits:
-              memory: 32Gi
-              gpu.intel.com/i915: "1"
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: ["ALL"]
-        - name: redis
-          # Its own, not the namespace's: two paperless instances sharing a
-          # celery broker would take each other's tasks.
-          image: redis:8.10.1-alpine
-          restartPolicy: Always
-          args: ["redis-server", "--save", "", "--loglevel", "warning"]
-          startupProbe:
-            tcpSocket:
-              port: 6379
-            periodSeconds: 2
-            failureThreshold: 30
-          resources:
-            requests:
-              cpu: 20m
-              memory: 64Mi
-            limits:
-              memory: 256Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-        - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          restartPolicy: Always
-          envFrom:
-            - configMapRef:
-                name: paperless-eval-config
-          env:
-            # Overrides the namespace ConfigMap: this pod's own llama-server,
-            # and its own redis rather than the shared Service.
-            - name: PAPERLESS_AI_LLM_ENDPOINT
-              value: http://127.0.0.1:8080/v1
-            - name: PAPERLESS_AI_LLM_MODEL
-              value: qwen3.6-35b-a3b
-            - name: PAPERLESS_REDIS
-              value: redis://127.0.0.1:6379
-            - name: PAPERLESS_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-eval-secrets
-                  key: secret-key
-            - name: PAPERLESS_DBUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PAPERLESS_DBPASS
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          ports:
-            - name: http
-              containerPort: 8000
-          volumeMounts:
-            - name: data
-              mountPath: /usr/src/paperless/data
-            - name: media
-              mountPath: /usr/src/paperless/media
-            - name: consume
-              mountPath: /usr/src/paperless/consume
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            failureThreshold: 180
-          # Same as production, and required: the entrypoint starts as root,
-          # chowns the volumes and drops to uid 1000 via setgid/setuid.
-          securityContext:
-            runAsNonRoot: false
-            allowPrivilegeEscalation: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              memory: 2Gi
-      containers:
-        - name: eval
-          # The paperless image again, for psycopg and a python that matches
-          # the application — not because the script needs paperless itself.
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          command: ["python3", "/eval/eval.py"]
-          env:
-            - name: EVAL_LABEL
-              value: qwen35
-            - name: EVAL_SAMPLE
-              value: "50"
-            - name: PGHOST
-              value: pprleval-pg-rw
-            - name: PGDATABASE
-              value: paperless
-            - name: PGUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PGPASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          volumeMounts:
-            - name: script
-              mountPath: /eval
-            - name: results
-              mountPath: /results
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 512Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: ["ALL"]
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llama-models
-        - name: results
-          persistentVolumeClaim:
-            claimName: paperless-eval-results
-        - name: script
-          configMap:
-            name: paperless-eval-script
-        # The namespace's /data, mounted read-only and copied out by seed-data
-        # above. Never mounted as the live /data of a job.
-        - name: golden
-          persistentVolumeClaim:
-            claimName: paperless-eval-golden
-            readOnly: true
-        # paperless insists on all three paths existing. media and consume stay
-        # empty — nothing is consumed during a measurement — while data is
-        # seeded with the copy.
-        - name: data
-          emptyDir: {}
-        - name: media
-          emptyDir: {}
-        - name: consume
-          emptyDir: {}