Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 364cc01291 fix(ci): deploy the proxies when their inventory changes, not just their roles
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Arming the bouncer on proxy-2 (#766) merged and deployed nothing. The switch
lives in ansible/inventory/host_vars/proxy-2.renner.uno.yml, and the trigger
list covered ansible/roles/** and the play but no inventory path at all. The
change sat in git looking applied while the host kept running dry-run, and
nothing reported a problem — there was simply no pipeline run to fail.

The bouncer's LAPI key has the same shape: it lives in
group_vars/netbird_proxy/vault.yml, so a rotation would never have reached the
hosts either.

Both paths are now listed. pve_hosts.yml deliberately is not: it is the single
inventory file for the whole fleet, so listing it would redeploy the proxies
for any unrelated host edit. The cost is that a change to [netbird_proxy] group
membership needs a manual deployment run.

Merging this triggers the run itself, since .woodpecker/deploy-netbird-proxy.yml
is in its own trigger list — so the arming from #766 applies with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:18:03 +02:00
k.renner c99e94cc0b Merge pull request 'feat(crowdsec): arm the bouncer on proxy-2' (#766) from arm-crowdsec-proxy-2 into master
Reviewed-on: k.renner/homelab#766
2026-09-02 14:15:42 +00:00
claude-botandClaude Opus 5 05cd7f3674 feat(crowdsec): arm the bouncer on proxy-2
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
First host goes from dry-run to enforcing. proxy-2 deliberately: it is the
dual-stack member, while proxy-1 is IPv6-only and the one where every
network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
and a GitHub download that could not work at all. host_vars keeps the change
to one host; proxy-1 stays on the role default.

Verified before committing: ansible-inventory resolves nftables for proxy-2
and nothing for proxy-1.

Adds netbird.renner.uno to capi_whitelists first, and that is not decoration.
The bouncer's chain sits on the INPUT hook, the proxy hosts run no other
firewall, and there is therefore no conntrack rule accepting established flows
ahead of it. A ban on the management address would drop the return traffic of
the proxies' own outbound control connection and take them off the mesh, with
no inbound attack involved at all. The same reasoning already covers the proxy
addresses, which were listed from the start.

Checked at the time of writing: none of these addresses appear in the current
decision set — searched all ~15k entries. That is the state today, not a
guarantee; they are rented addresses on a hoster whose ranges do end up on
community blocklists, which is the whole reason for listing them.

The delivery path was tested end to end beforehand with a decision on
192.0.2.1 (TEST-NET-1, so no real traffic could be affected). Both bouncers
received the ban within one poll cycle — a 206-byte delta against the usual
26 — and the revocation the same way, 203 bytes, ~5s after deletion. What that
did not test is whether a packet is actually dropped, which dry-run cannot
show. This commit is what makes that testable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:13:11 +02:00
k.renner d655d65ad9 Merge pull request 'feat(llama-server): 32k context per slot for the Assist agent' (#765) from llama-ctx-assist into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#765
2026-09-02 14:09:07 +00:00
claude-botandClaude Opus 5 60a3da9447 feat(llama-server): 32k context per slot for the Assist agent
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The Home Assistant conversation agent is configured and refuses every
request:

    API error: request (18910 tokens) exceeds the available
    context size (16384 tokens), try increasing it

18910 breaks down as ~12k for the exposed-entity overview and ~7k for
the base instructions and tool schemas, whose parameters enumerate every
exposed name, area and floor. Trimming the exposed set from 845 entities
to 666 already took the overview from 16045 tokens to 12048; the
remainder is not the sort of thing curation reaches.

--ctx-size is per slot rather than the total, which is what makes this
affordable: /props reports n_ctx_slot equal to the flag while running
four slots, where a total would have divided down to 4096. Doubling the
flag alone would double the KV cache, so the slot count halves in step
and the allocation lands where it already is -- 24979 MiB against the
32Gi limit, most of it the 19.5 GiB of weights.

Two slots matches the actual callers: one holds Home Assistant's prompt,
one holds Paperless's, and prompt caching keeps each warm. A cold prefill
of this prompt costs about two minutes, so a third caller queueing is a
better outcome than it evicting one of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:07:06 +02:00
k.renner 47086a5fe9 Merge pull request 'feat(monitoring): scrape CrowdSec and give it a dashboard' (#764) from crowdsec-metrics-dashboard into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#764
2026-09-02 13:59:21 +00:00
k.renner 412acf7cb4 Merge pull request 'feat(network-policies): let Home Assistant reach the chat model' (#759) from ha-llm-agent into master
Reviewed-on: k.renner/homelab#759
2026-09-02 13:59:00 +00:00
k.renner 75ecf020cc Merge pull request 'fix(llama-embed): size the batch for bge-m3 tokenizer, not Paperless' (#762) from embed-batch-fix into master
Reviewed-on: k.renner/homelab#762
2026-09-02 13:58:42 +00:00
claude-botandClaude Opus 5 0a101eb8ee fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The LLM index rebuild ran for 79 minutes and then died:

    input (1069 tokens) is too large to process.
    increase the physical batch size (current batch size: 1024)

Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.

1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.

The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:13:54 +02:00
claude-botandClaude Opus 5 7903c81c9a feat(network-policies): let Home Assistant reach the chat model
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
HA 2026.8.3 ships a native `llama_cpp` integration, so an Assist
conversation agent backed by the cluster's own model needs no custom
component — only a path from the HA pod to llama-server, which two
default-deny policies currently block in both directions. Verified from
inside the pod: the call to llama-server.ai.svc.cluster.local:8080 times
out today.

Both grants are scoped to the single pod and port, mirroring the
paperless grant next to them, so neither widens if something else is
placed in the other namespace. llama-embed is deliberately excluded: the
conversation agent uses chat completions only.

Tool calling was verified against the running model before adding this —
the Assist API requires it, and llama.cpp only exposes it for some
builds. b10731 returns finish_reason=tool_calls and parses German
commands into the right arguments without --jinja.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 14:18:30 +02:00
7 changed files with 108 additions and 12 deletions
+12
View File
@@ -10,6 +10,18 @@ when:
# install-netbird-proxy.yml also applies the edge bouncer.
- "ansible/roles/crowdsec-edge-bouncer/**"
- "ansible/plays/install-netbird-proxy.yml"
# The variables that steer those roles, not just the roles themselves.
# Without these two, a change that only touches inventory never deploys
# and nothing says so: arming the bouncer lives in host_vars, and its
# LAPI key lives in group_vars, so both would sit in git looking applied
# while the hosts kept running the previous configuration.
#
# pve_hosts.yml is deliberately NOT listed. It is the single inventory
# file for the whole fleet, so listing it would redeploy the proxies for
# any unrelated host edit. The cost is that a change to [netbird_proxy]
# group membership needs a manual deployment run.
- "ansible/inventory/host_vars/proxy-*.yml"
- "ansible/inventory/group_vars/netbird_proxy/**"
- ".woodpecker/deploy-netbird-proxy.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
@@ -0,0 +1,19 @@
---
# Arms the CrowdSec firewall bouncer on THIS host only.
#
# The role defaults to dry-run for the whole netbird_proxy group, so the two
# members can be armed one at a time. proxy-2 goes first deliberately: it is
# the dual-stack member, while proxy-1 is IPv6-only and the one where every
# network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
# and the GitHub download that could not work at all.
#
# What flipping this does: the bouncer stops logging what it would block and
# creates two nftables tables (ip crowdsec, ip6 crowdsec6) with the decisions
# it holds. It touches no existing chain and sets no chain policy.
#
# Rollback is `systemctl stop crowdsec-firewall-bouncer` — a clean stop deletes
# both tables itself. Setting this back to dry-run and re-running the play does
# the same through the handler.
#
# Remove this file once proxy-1 follows and the role default moves to nftables.
crowdsec_bouncer_mode: nftables
+20 -4
View File
@@ -75,11 +75,27 @@ spec:
# many small per-expert matmuls that sets the pace, not bandwidth.
- --threads
- "4"
# 16k rather than 8k because the measured prefill falloff is mild
# and this model's KV cache is unusually cheap — only 10 of its 40
# layers use full attention.
# 32k because Home Assistant's Assist agent measured 18910 tokens
# for a single request: ~12k of exposed-entity overview plus ~7k of
# base instructions and tool schemas, whose parameters enumerate
# every exposed name, area and floor. At 16384 the request was
# refused outright.
#
# --ctx-size is per slot, not the total: /props reports
# n_ctx_slot equal to this value while running four slots, where a
# total would have divided down to 4096. Doubling it alone would
# therefore double the KV cache, so the slot count halves in step
# and the allocation stays where it is today — 24979 MiB of the
# 32Gi limit, most of it the 19.5 GiB of weights.
#
# Two slots is what is actually in use: one holds Home Assistant's
# prompt, one holds Paperless's. A third caller would queue rather
# than evict either, which is the better failure mode given a cold
# prefill of this prompt costs around two minutes.
- --ctx-size
- "16384"
- "32768"
- --parallel
- "2"
# A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
# with ErrorDeviceLost.
- --ubatch-size
+19 -8
View File
@@ -64,16 +64,27 @@ spec:
- "99"
- --threads
- "2"
# Paperless chunks documents at 1024 tokens. The defaults are
# batch 2048 / ubatch 512, and with --embeddings llama-server
# clamps both down to 512 rather than run them mismatched — below
# the chunk size. Matching both to 1024 fits a whole chunk in one
# pass; 2048 is deliberately avoided, since that is where this
# Vulkan stack fails with ErrorDeviceLost.
# Paperless chunks at 1024 tokens, but counts them with its own
# tokenizer while this server measures the batch in bge-m3's.
# Those disagree: XLM-RoBERTa splits German into more pieces than
# llama_index's counter does, so a chunk Paperless considers full
# at 1024 arrived here as 1069 and the request failed outright —
# "input (1069 tokens) is too large to process".
#
# The headroom therefore has to absorb a tokenizer ratio, not just
# a rounding error, and 1.5x covers far more than the 1.044 that
# broke it. The defaults are batch 2048 / ubatch 512, and with
# --embeddings llama-server clamps both to the smaller rather than
# run them mismatched, so both are set.
#
# 2048 stays avoided: that is where the chat model met
# ErrorDeviceLost on this Vulkan stack. That failure came under a
# 20 GB model's memory pressure and may not bind a 600 MB one, but
# this does not need to find out.
- --batch-size
- "1024"
- "1536"
- --ubatch-size
- "1024"
- "1536"
# Room for a full chunk plus overhead rather than exactly one.
- --ctx-size
- "4096"
@@ -65,6 +65,21 @@ spec:
- 45.157.177.26/32
- 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
- 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
# netbird.renner.uno — management, signal, relay and TURN.
#
# Not optional once a bouncer is armed. Its chain sits on the INPUT
# hook, the proxy hosts run no other firewall, and there is therefore
# no conntrack rule accepting established flows ahead of it. A ban on
# this address would drop the RETURN traffic of the proxies' own
# outbound control connection and take them off the mesh — with no
# inbound attack involved at all.
#
# Checked when this was added: none of these addresses appear in the
# current decision set. That is the state today, not a guarantee —
# these are rented addresses on a hoster whose ranges do end up on
# community blocklists.
- 162.55.50.97/32
- 2a01:4f8:c2c:4579::1/128
# mTLS between agents and the LAPI. Do not switch back to password auth:
# the LAPI runs without persistence, so its machine-registration DB is
@@ -34,6 +34,17 @@ spec:
- ports:
- port: "8080"
protocol: TCP
# Home Assistant's llama.cpp conversation agent calls the chat model only —
# it has no use for the embedding endpoint, so unlike paperless above this
# grant does not extend to llama-embed.
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": home-assistant
app.kubernetes.io/name: home-assistant
toPorts:
- ports:
- port: "8080"
protocol: TCP
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
@@ -167,6 +167,18 @@ spec:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": influxdb
# The cluster's own chat model, for the llama.cpp conversation agent that
# backs Assist. Scoped to the one pod and port rather than the ai namespace
# as a whole, so this grant cannot widen if something else is placed there —
# the embedding endpoint next to it is deliberately not included.
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ai
app.kubernetes.io/name: llama-server
toPorts:
- ports:
- port: "8080"
protocol: TCP
# traefik for east-west OIDC roundtrips that go through the
# public hostname via k8s_gateway.
- toEndpoints: