konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Restore test 6/9: hardware-bound workloads run where the node declares the hardware #1302

Closed
claude-bot wants to merge 2 commits from split/06-hardware-gating into split/05-infra-services
pull from: split/06-hardware-gating
merge into: :split/05-infra-services
Conversation 2 Commits 2 Files Changed 26
+98 -48
claude-bot commented 2026-10-17 09:52:04 +00:00
Member
Copy Link
Copy Source

Part 6 of 9 (stacked on part 5). Workloads bound to hardware run only where a node declares it (labels from part 2):

  • hardware.renner.uno/intel-gpu: intel-gpu-plugin and exporter DaemonSets, llama-server, llama-embed, wiki-embed, wiki.
  • net.renner.uno/iot-vlan: the macvlan workloads on v30iot (shelly-manager, mDNS relay, matter-server, music-assistant, otbr).
  • shelly-manager and the mDNS relay move from home-assistant into home-assistant-iot (depends on home-assistant), so HA and what depends on it do not wait on the IoT VLAN. The shelly-manager-auth ExternalSecret stays in home-assistant: mqtt-account-controller mounts it too.

Rollout: labels on the nodes first (part 2), or these pods go Pending. Moving shelly-manager/mDNS relay between Kustomizations is two-phase; shelly-manager's PVC already carries prune: disabled.

🤖 Generated with Claude Code

Part 6 of 9 (stacked on part 5). Workloads bound to hardware run only where a node declares it (labels from part 2): - `hardware.renner.uno/intel-gpu`: intel-gpu-plugin and exporter DaemonSets, llama-server, llama-embed, wiki-embed, wiki. - `net.renner.uno/iot-vlan`: the macvlan workloads on v30iot (shelly-manager, mDNS relay, matter-server, music-assistant, otbr). - shelly-manager and the mDNS relay move from `home-assistant` into `home-assistant-iot` (depends on home-assistant), so HA and what depends on it do not wait on the IoT VLAN. The `shelly-manager-auth` ExternalSecret stays in home-assistant: mqtt-account-controller mounts it too. **Rollout:** labels on the nodes first (part 2), or these pods go Pending. Moving shelly-manager/mDNS relay between Kustomizations is two-phase; shelly-manager's PVC already carries `prune: disabled`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 2 commits 2026-10-17 09:52:04 +00:00
feat(gpu): iGPU workloads run only where a node declares the iGPU 56d150451c 
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are bound to the Intel iGPU. On a site without one (Hetzner,
restore rehearsal runs 27/28) their pods stayed Pending, and the GPU plugin
and exporter DaemonSets sat on every node with nothing to serve.

The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki now select on it. Elsewhere the
DaemonSets have no pods and the Deployments stay unscheduled.

Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(iot): v30iot workloads run only where a node declares the IoT VLAN; HA no longer waits on them
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
review-bot waiting to retry: HTTP 429, retry 1/3 in 1m0s
Details
a5a1f43c76 
Five workloads attach a macvlan on ens19 (v30iot) with DHCP IPAM:
shelly-manager and the mDNS relay (inside the home-assistant Kustomization),
matter-server, music-assistant and otbr. On a node without that VLAN the CNI
add fails and the pod never starts.

- k8s-1..3 carry net.renner.uno/iot-vlan: "true"; each of them declares
  ens19 in its VLANConfig. The five Deployments select on it. The comments
  they replace described the retired k8s-edge taint.
- shelly-manager and the mDNS relay move to apps/home-assistant-iot, with a
  Kustomization of their own (depends on gitea, home-assistant and the
  infrastructure Kustomizations). home-assistant's wait: true no longer
  covers them, so HA and everything waiting on it (evcc, trmnl,
  zone-configurator, speech-to-phrase) are Ready without the VLAN, and a
  broken IoT bridge no longer holds them back in production either.
- shelly-manager-auth stays in apps/home-assistant: mqtt-account-controller
  mounts it too, and that Kustomization comes first. Moved with
  shelly-manager, mqtt-account-controller would wait for a Secret that only
  the later Kustomization creates.
- The probes-blackbox and authentik blueprint comments name the new paths.

On a site without the label the IoT Kustomizations fail at their timeout,
naming the Pending pods; nothing else depends on them.

Production rollout:
1. Label the nodes first (talosctl patch the nodeLabels), or the five pods go
   Pending.
2. The move between Kustomizations is two-phase (add under
   home-assistant-iot, then drop from home-assistant). shelly-manager's PVC
   already carries prune: disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model mistral-large-4-0, commit a5a1f43c762230b61b60dfdeffa1eb0507324b5d.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

Part 6 of the 9-part restore-test stack. It gates hardware-bound workloads on the node labels that part 2 (already on the base branch, commit d6a9575) declares in kubernetes/talos/patches/k8s-{1,2,3}.yaml: hardware.renner.uno/intel-gpu on the intel-gpu-plugin DaemonSet, intel-gpu-exporter, llama-server, llama-embed, wiki-embed and wiki; net.renner.uno/iot-vlan on the five v30iot macvlan workloads (shelly-manager, mDNS relay, matter-server, music-assistant, otbr). It also moves shelly-manager and the mDNS relay from apps/home-assistant into a new apps/home-assistant-iot Kustomization with its own Flux Kustomization (dependsOn: home-assistant), so HA and its dependents (evcc, trmnl, zone-configurator, speech-to-phrase) no longer wait on the IoT VLAN. I verified the labels exist on all three production nodes in the base branch, that the gating list is complete (whisper, speech-to-phrase, openwakeword and immich-ML are CPU-only; HA and esphome deliberately have no macvlan; the only v30iot NADs are the five gated ones), that nothing else in the Flux graph depends on the moved workloads, and that the moved Services carry no pinned ClusterIPs and the PVC is prune: disabled. CI (ci-manifests-test, ci-scripts-test) is green on the head SHA; the kustomize-build step discovers the new directory automatically.

⚪ Unverified

  • Whether the production nodes already carry the two labels. Part 2's rollout step is a talosctl patch (no reboot), which is not observable from the repo. If this PR merges before the labels are patched, the gated pods go Pending on production (and the scheduled e2e "no failing pods" check would flap) until the patch lands. The PR documents this ordering, but I could not confirm the labels are live.
  • The live-cluster behavior of the Kustomization move (prune-then-recreate gap for shelly-manager/mDNS relay) is reasoned from Flux semantics, not observed.

🟡 Suggestions

  • The commit message says the move between Kustomizations "is two-phase (add under home-assistant-iot, then drop from home-assistant)", but this PR implements both halves in one commit. Merged as-is, the home-assistant Kustomization prunes the shelly-manager/mDNS-relay objects during its reconcile and home-assistant-iot (new Kustomization at kubernetes/clusters/homelab/home-assistant.yaml:61-84, entries removed from kubernetes/apps/home-assistant/kustomization.yaml) recreates them afterwards — a minutes-long outage of shelly-manager and the mDNS relay on the live cluster. The end state is safe (PVC prune: disabled at kubernetes/apps/home-assistant-iot/shelly-manager-pvc.yaml:5, no pinned Service IPs, ConfigMap recreated with the pod), but splitting this into two PRs would avoid the gap entirely.

🟢 Nits

  • kubernetes/infrastructure/intel-gpu-plugin/daemonset.yaml:27-29 still says "no nodeSelector. Every node carries the same iGPU, leaving nothing to discover; the plugin exits cleanly on a node without /dev/dri", which now contradicts the nodeSelector: hardware.renner.uno/intel-gpu: "true" added at line 122. The comment should describe the label gating instead.

Tests

Adequate for this repo's conventions. The head SHA passes ci-manifests-test (kustomize build of every kustomization — the new home-assistant-iot directory is picked up by the find-based discovery — plus woodpecker-lint, shellcheck, promtool and yaml-citation checks) and ci-scripts-test. There are no manifest unit tests beyond that; the restore rehearsal (part 9) is the integration test. Optional hardening for future gating changes: a CI check that every nodeSelector label used under kubernetes/apps and kubernetes/infrastructure is declared in kubernetes/talos/patches — it would have caught a label/patch mismatch mechanically.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `mistral-large-4-0`, commit `a5a1f43c762230b61b60dfdeffa1eb0507324b5d`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary Part 6 of the 9-part restore-test stack. It gates hardware-bound workloads on the node labels that part 2 (already on the base branch, commit `d6a9575`) declares in `kubernetes/talos/patches/k8s-{1,2,3}.yaml`: `hardware.renner.uno/intel-gpu` on the intel-gpu-plugin DaemonSet, intel-gpu-exporter, llama-server, llama-embed, wiki-embed and wiki; `net.renner.uno/iot-vlan` on the five v30iot macvlan workloads (shelly-manager, mDNS relay, matter-server, music-assistant, otbr). It also moves shelly-manager and the mDNS relay from `apps/home-assistant` into a new `apps/home-assistant-iot` Kustomization with its own Flux Kustomization (`dependsOn: home-assistant`), so HA and its dependents (evcc, trmnl, zone-configurator, speech-to-phrase) no longer wait on the IoT VLAN. I verified the labels exist on all three production nodes in the base branch, that the gating list is complete (whisper, speech-to-phrase, openwakeword and immich-ML are CPU-only; HA and esphome deliberately have no macvlan; the only v30iot NADs are the five gated ones), that nothing else in the Flux graph depends on the moved workloads, and that the moved Services carry no pinned ClusterIPs and the PVC is `prune: disabled`. CI (ci-manifests-test, ci-scripts-test) is green on the head SHA; the kustomize-build step discovers the new directory automatically. ## ⚪ Unverified - Whether the production nodes already carry the two labels. Part 2's rollout step is a `talosctl patch` (no reboot), which is not observable from the repo. If this PR merges before the labels are patched, the gated pods go Pending on production (and the scheduled e2e "no failing pods" check would flap) until the patch lands. The PR documents this ordering, but I could not confirm the labels are live. - The live-cluster behavior of the Kustomization move (prune-then-recreate gap for shelly-manager/mDNS relay) is reasoned from Flux semantics, not observed. ## 🟡 Suggestions - The commit message says the move between Kustomizations "is two-phase (add under home-assistant-iot, then drop from home-assistant)", but this PR implements both halves in one commit. Merged as-is, the `home-assistant` Kustomization prunes the shelly-manager/mDNS-relay objects during its reconcile and `home-assistant-iot` (new Kustomization at `kubernetes/clusters/homelab/home-assistant.yaml:61-84`, entries removed from `kubernetes/apps/home-assistant/kustomization.yaml`) recreates them afterwards — a minutes-long outage of shelly-manager and the mDNS relay on the live cluster. The end state is safe (PVC `prune: disabled` at `kubernetes/apps/home-assistant-iot/shelly-manager-pvc.yaml:5`, no pinned Service IPs, ConfigMap recreated with the pod), but splitting this into two PRs would avoid the gap entirely. ## 🟢 Nits - `kubernetes/infrastructure/intel-gpu-plugin/daemonset.yaml:27-29` still says "no nodeSelector. Every node carries the same iGPU, leaving nothing to discover; the plugin exits cleanly on a node without /dev/dri", which now contradicts the `nodeSelector: hardware.renner.uno/intel-gpu: "true"` added at line 122. The comment should describe the label gating instead. ## Tests Adequate for this repo's conventions. The head SHA passes `ci-manifests-test` (kustomize build of every kustomization — the new `home-assistant-iot` directory is picked up by the find-based discovery — plus woodpecker-lint, shellcheck, promtool and yaml-citation checks) and `ci-scripts-test`. There are no manifest unit tests beyond that; the restore rehearsal (part 9) is the integration test. Optional hardening for future gating changes: a CI check that every `nodeSelector` label used under `kubernetes/apps` and `kubernetes/infrastructure` is declared in `kubernetes/talos/patches` — it would have caught a label/patch mismatch mechanically. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=a5a1f43c762230b61b60dfdeffa1eb0507324b5d -->
claude-bot commented 2026-10-17 20:25:45 +00:00
Author
Member
Copy Link
Copy Source

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).
claude-bot closed this pull request 2026-10-17 20:25:45 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1302