feat(multus): add whereabouts IPAM for segments without a DHCP server #637
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#637
Reference in New Issue
Block a user
Step 1 of 3 toward moving Ceph off host networking. Lands inert — nothing references whereabouts yet, no existing attachment changes, and the installed CNI binary is not invoked until a NetworkAttachmentDefinition names it.
Why a third IPAM. The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by design. Neither option already present covers it:
dhcp(cni-dhcp-daemon)host-local(cni-plugins-installer)Whereabouts coordinates through the Kubernetes API; allocations live in the
ippools/overlappingrangeipreservationsCRDs. Nothing about assigning an address touches the segment being addressed — that is the property that makes it usable on an isolated SAN.On hostNetwork + privileged. The DaemonSet carries both. That is upstream's shape and it matches the three CNI components already here —
multus-daemonset.yaml,cni-dhcp-daemon.yaml,cni-plugins-installer.yamlall do the same. The pod writes a CNI binary and config into/opt/cni/binand/etc/cni/net.d; that is node plumbing, and it has to work before the attachment it serves exists. Fourth instance of an existing exception, not a new one — flagged explicitly rather than filed under "already the case".The reconciler is split out and needs neither: it only talks to the API, so it runs as an ordinary pod-network Deployment. It matters more than it looks — the normal release path is the CNI DEL when a pod goes away, which does not run when a node is lost hard, and on a
/29a few leaked addresses decide whether an OSD gets one.Provenance. Vendored from upstream v0.9.4, image tag pinned (upstream ships
:latest). Tag not digest, matching the neighbouring images. CRDs vendored unmodified so a future upstream diff stays readable.nodeslicepoolsis included because the ClusterRole references it; the node-slice controller itself is not deployed.Verified: kustomize builds; all nine resources pass a server-side dry run.
Next: NAD for the SAN, then the CephCluster provider switch plus mon failovers.
Groundwork for moving Ceph's cluster network off host networking. Nothing references whereabouts yet, so this lands inert: no existing attachment changes, and the DaemonSet only installs a CNI binary that nothing invokes until a NetworkAttachmentDefinition names it. The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by design. Neither IPAM already available covers it: dhcp cni-dhcp-daemon works where a DHCP server is on the wire. There is none on the SAN and there must not be one. host-local installed by cni-plugins-installer, but it keeps allocation state per node with no cluster-wide coordination. On a shared L2 segment two nodes hand out the same address. Whereabouts coordinates through the Kubernetes API — allocations live in the ippools and overlappingrangeipreservations CRDs. Nothing about assigning an address touches the segment being addressed, which is the property that makes it usable on an isolated SAN. The DaemonSet runs hostNetwork and privileged. That is upstream's shape, and it matches what multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml already do: the pod writes a CNI binary and its config into /opt/cni/bin and /etc/cni/net.d, which is node plumbing, and it has to work before the attachment it serves exists. It is the fourth instance of an existing exception, not a new one. The reconciler is split out deliberately. It needs no host access — only the API, where the allocations are — so it runs as an ordinary pod-network Deployment. It matters more than it looks: the normal release path is the CNI DEL when a pod goes away, which does not run when a node is lost hard, and on a /29 a handful of leaked addresses is the difference between an OSD getting one and not. Vendored from upstream v0.9.4 with the image tag pinned — upstream ships `:latest`. Tag rather than digest, matching the other images in this directory. The three CRDs are vendored unmodified so a future upstream diff stays readable. nodeslicepools is included because the ClusterRole references it, though the node-slice controller itself is not deployed. Verified: kustomize builds; all nine resources pass a server-side dry run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commitebd962592658fcc3c8a333873fc6fa9c9b2d0362.Summary
Adds the whereabouts CNI IPAM plugin (CRDs, a node DaemonSet, and a reconciler Deployment) to the
multuskustomization. As the PR body states, this is genuinely inert — no NetworkAttachmentDefinition referenceswhereaboutsyet, so none of this is exercised by any running workload today. The manifests are vendored from upstream v0.9.4 with the image tag pinned. Overall this is a clean, well-documented vendoring PR; I found one real (if currently dormant) manifest inconsistency and a couple of things worth a second look, no outright blocking defects.🟡 Suggestions
kubernetes/infrastructure/multus/whereabouts-daemonset.yaml:401-403andwhereabouts-reconciler.yaml:493-495— both new workloads tolerate onlyNoSchedule, whereas all three existing CNI components in this directory (multus-daemonset.yaml:159-163,cni-dhcp-daemon.yaml:64-68,cni-plugins-installer.yaml:34-38) tolerate bothNoScheduleandNoExecute. The PR's own rationale ("Fourth instance of an existing exception, not a new one") explicitly claims parity with those three, but the toleration set doesn't actually match. Practical effect: on this cluster every node is control-plane and presumably carries aNoScheduletaint (handled), but if a node is later markedNotReady/unreachable(kubelet hiccup, brief network blip), theDefaultTolerationSecondsadmission plugin only grants an implicit 300s grace period forNoExecutebefore evicting whereabouts — unlike the sibling CNI daemons, which tolerate it indefinitely and stay put. For node-plumbing pods that are supposed to survive exactly this kind of instability (per the DaemonSet's own header comment), that's the wrong default. Worth adding the sameNoExecutetoleration for consistency, even though it has no observable effect while the deployment stays inert.whereabouts-daemonset.yaml:406-412— the container entrypoint runs/ip-control-loop -log-level debugas the last statement of a/bin/sh -c "..."script, not viaexec. If that's how upstream'sdoc/crds/daemonset-install.yamlactually does it, this is fine (harmless either way since Kubernetes kills the whole cgroup on termination), but it's worth a quick sanity check against the vendored source:sh -cscripts that don'texectheir final command mean SIGTERM sent to PID 1 is not forwarded toip-control-loopby default, so shutdown falls back to waiting outterminationGracePeriodSeconds(unset here, so the Kubernetes default of 30s) before SIGKILL rather than a graceful stop. Not something I could verify without fetching the actual upstream file, so treat this as a "confirm against source" item rather than a confirmed bug.whereabouts-daemonset.yaml:405-432lackspriorityClassName, whilecni-dhcp-daemon.yamlsetssystem-node-criticalfor the same class of "must come up before the plugin it fronts can be invoked" reasoning this PR uses to justify hostNetwork/privileged. (Notemultus-daemonset.yamlandcni-plugins-installer.yamlalso omit it, so the existing convention here is already inconsistent — this isn't a regression, just an omission worth considering if these pods are meant to be as critical as the header comment implies.)Tests
No automated tests exist for this directory (none of the sibling CNI manifests have any either), so nothing is missing relative to repo convention. The PR body states kustomize build and a server-side dry run were done manually; I don't have
kustomize/kubectlavailable in this environment to independently re-verify that claim, but I did check for RBAC/naming collisions with existing cluster resources (none — seewhereabouts/whereabouts-cni/whereabouts-config/whereabouts-reconcilernames, all new) and confirmed the parentkubernetes/infrastructure/kustomization.yaml:9reference tomultus/needs no changes since it already points at the directory rather than individual files.Recommendation
Looks reasonable — no blocking issues; comments above are optional (the toleration point is the one worth actually fixing before this stops being inert).