Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 0a101eb8ee fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The LLM index rebuild ran for 79 minutes and then died:

    input (1069 tokens) is too large to process.
    increase the physical batch size (current batch size: 1024)

Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.

1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.

The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:13:54 +02:00
3 changed files with 23 additions and 34 deletions
+19 -8
View File
@@ -64,16 +64,27 @@ spec:
- "99"
- --threads
- "2"
# Paperless chunks documents at 1024 tokens. The defaults are
# batch 2048 / ubatch 512, and with --embeddings llama-server
# clamps both down to 512 rather than run them mismatched — below
# the chunk size. Matching both to 1024 fits a whole chunk in one
# pass; 2048 is deliberately avoided, since that is where this
# Vulkan stack fails with ErrorDeviceLost.
# Paperless chunks at 1024 tokens, but counts them with its own
# tokenizer while this server measures the batch in bge-m3's.
# Those disagree: XLM-RoBERTa splits German into more pieces than
# llama_index's counter does, so a chunk Paperless considers full
# at 1024 arrived here as 1069 and the request failed outright —
# "input (1069 tokens) is too large to process".
#
# The headroom therefore has to absorb a tokenizer ratio, not just
# a rounding error, and 1.5x covers far more than the 1.044 that
# broke it. The defaults are batch 2048 / ubatch 512, and with
# --embeddings llama-server clamps both to the smaller rather than
# run them mismatched, so both are set.
#
# 2048 stays avoided: that is where the chat model met
# ErrorDeviceLost on this Vulkan stack. That failure came under a
# 20 GB model's memory pressure and may not bind a 600 MB one, but
# this does not need to find out.
- --batch-size
- "1024"
- "1536"
- --ubatch-size
- "1024"
- "1536"
# Room for a full chunk plus overhead rather than exactly one.
- --ctx-size
- "4096"
@@ -108,24 +108,6 @@ spec:
lapi:
replicas: 1
service:
# Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
#
# These belong on the SERVICE, not on the Ingress: under the Kubernetes
# Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
# annotations, and on an Ingress they are silently ignored. The symptom
# is Traefik speaking plaintext to a TLS-only backend, which the LAPI
# answers with "Client sent an HTTP request to an HTTPS server." and a
# 400 — an error that looks like a malformed request rather than a
# scheme mismatch.
#
# With tls.enabled the LAPI serves HTTPS on 8080 and offers no
# plaintext port, so https here is mandatory rather than a hardening
# choice. The ServersTransport carries the chart's own CA, since that
# certificate is self-signed.
annotations:
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
# Far below the chart default, which reserves half a core per LAPI for a
# workload that draws single-digit millicores — enough to leave the
# cluster unschedulable with one node out. Only the reservation shrinks;
@@ -44,8 +44,8 @@ metadata:
namespace: crowdsec
spec:
serverName: crowdsec-service.crowdsec.svc.cluster.local
rootCAs:
- secret: crowdsec-root-secret
rootCAsSecrets:
- crowdsec-root-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
@@ -60,12 +60,8 @@ metadata:
cert-manager.io/cluster-issuer: letsencrypt-cluster
# NOT behind the Authentik gate: the client is a daemon with an API key,
# not a browser that can complete an OIDC flow.
#
# The backend scheme and ServersTransport are NOT set here. Under the
# Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
# are SERVICE annotations — on an Ingress they are silently ignored, and
# Traefik then talks plaintext to a TLS-only backend. They live on the
# chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
spec:
ingressClassName: traefik
rules: