claude-bot
1f18599b4c
Both blocking findings from the review on #709, and both were real. The deleted cilium-cni.yaml was the only place in the repo that set cluster.network.cni.name: none and cluster.proxy.disabled: true. They sat in its second document next to the inline manifest, so removing the file removed them too — a fresh `talosctl gen config` from this repo's patches would have produced a config with Talos's default CNI and kube-proxy running, against a Cilium release configured with kubeProxyReplacement: true. Two runbooks meanwhile asserted that cluster.yaml supplied both, which it did not. RUNBOOK-baremetal-conversion.md also still passed `@<cilium additive half>` in its gen-config example, three lines above prose saying there is no CNI patch any more. The equivalent block in the cold-rebuild runbook had been fixed; this one was missed. Verified the way it should have been in the first place — by running the runbook's own command rather than inspecting the live cluster: talosctl gen config homelab https://k8s.renner.uno:6443 \ --output-types controlplane \ --config-patch @talos/patches/cluster.yaml \ --config-patch @talos/patches/registry-mirror.yaml \ --config-patch @talos/patches/k8s-1.yaml -> cni=none proxy.disabled=true -> 0 cilium manifests The live cluster could never have caught this. It has carried both settings since it was built, and the node configs were patched by hand rather than regenerated, so nothing in the verification touched the path a rebuild actually uses.
4.7 KiB
Cilium: one owner
The rule
Cilium is owned by Flux, through infrastructure/cilium/helmrelease.yaml.
Talos provides the conditions for a CNI and nothing else. Never both.
Talos-only would also be a valid answer — it is the simpler one, and it keeps a cold rebuild self-forming. It is rejected here only because it costs Flux's reconciliation, image automation and value ergonomics for the one component most likely to need a careful, reviewed change.
Where we are
Both own it, which is the state this document exists to end.
- 22 Cilium objects are listed in the
talos-bootstrap-manifests-inventoryConfigMap: both DaemonSets, the operator Deployment, both ConfigMaps, the ServiceAccounts, the ClusterRoles/Bindings and thecilium-secretsnamespace. That is the whole installation, not a fragment. - The same objects carry
meta.helm.sh/release-name: cilium. cilium-confighas three field managers:helm-controller,talosandkubectl-patch.
The consequences are not theoretical. The seed sat at Cilium 1.19.2 for twelve
days while the chart was on 1.20.1, and upgrade-k8s --dry-run shows that a
Kubernetes upgrade in that state downgrades the CNI and switches off socket
load balancing. Separately, enable-bgp-control-plane and the two
native-routing CIDRs exist in no file at all — applied by hand, so a cold
rebuild would have come up with BGP off while the peering CRDs in git made it
look configured.
Nothing fails when these drift. That is the whole problem.
Target state
Talos keeps exactly this, in talos/patches/cluster.yaml, and gains nothing
else:
cluster:
network:
cni:
name: none
proxy:
disabled: true
No inlineManifests entry for cilium. talos/patches/cilium-cni.yaml is
deleted.
Flux owns everything else. helmrelease.yaml is the single source for
every Cilium setting, including the three that were hand-patched.
A cold rebuild installs Cilium once, by hand, between talosctl bootstrap
and the Flux phase, from the same HelmRelease values. This is the cost of the
decision and it is deliberate: the cluster no longer forms itself from the
machine config alone. Given bootstrap.sh talos is already dead and Step 2 of
RUNBOOK-cold-rebuild.md is already a manual procedure, the marginal cost is one
documented command.
Transition
The order matters, because Talos prunes what its inventory used to list and no longer does — removing the manifest naively deletes the CNI.
-
Track the untracked settings first.
bgpControlPlane.enabled,ipv4NativeRoutingCIDR,ipv6NativeRoutingCIDRinto the HelmRelease. Note this is not a pure change of ownership: enabling bgpControlPlane through the chart also emits five companion keys the hand-patch never set, so Cilium has been running on its own defaults for router-id allocation. Verify the BGP sessions after Flux applies it. -
Remove cilium from
cluster.inlineManifests. This has no immediate effect. Evidence: Talos was upgraded to 1.13.9 four days ago and Cilium stayed at 1.20.1 — an OS upgrade does not reconcile bootstrap manifests. Onlyupgrade-k8sdoes. -
Prove the prune before running it.
upgrade-k8s --dry-run --manifests-no-pruneand confirm the 22 objects are not listed for deletion.NoPruneis honoured by the dry run as well as the apply (pkg/cluster/kubernetes/talos_managed.go), so this is a real test and not a reading of the flag description. -
Run
upgrade-k8s --manifests-no-prune. The inventory is rewritten without Cilium and the objects survive. Helm is then sole owner. This is the same run that takes Kubernetes 1.35.2 → 1.36.3. -
Verify the inventory lists no Cilium, the agents did not restart, and the BGP sessions are still established.
-
Delete
talos/patches/cilium-cni.yamland add the install step to RUNBOOK-cold-rebuild.md.
Not part of this, and not a defect either
The cluster runs routingMode: tunnel (the chart default, set nowhere) while
also advertising pod CIDRs over BGP. That is not one layer too many — the two
carry different directions:
- VXLAN carries pod-to-pod between nodes, directly, node to node.
- BGP gives the UDM routes into the pod CIDRs so traffic from the LAN can reach pods. That path goes through the router by definition.
Dropping the encapsulation in favour of native routing would not simplify anything; it would push inter-node pod traffic onto the underlay path and through the UDM, replacing a direct node-to-node hop with a router hop.
routingMode being unset is still worth knowing — the current behaviour is a
default nobody chose rather than a decision anyone recorded — but the default
is the right one here, and this document is not the place to revisit it.