feat(talos): move the node IP and etcd to pod0 (#576)

Pod east-west has to leave the management VLAN, and the node IP is the
only thing that controls where it goes. Three tests established that,
rather than argument:

  - --ipv4-node / --ipv6-node, applied via CiliumNodeConfig: the agent
    logged receiving both and still published its mgmt0 address.
  - directRoutingDevice=pod0: the agent logged "Direct routing device
    detected direct-routing-device=pod0" and still installed
    10.244.2.0/24 via 10.2.40.138 dev mgmt0.
  - Both routing modes, both address families, always keyed to the
    peer's Kubernetes InternalIP.

etcd is pinned to the same segment. Left unset it advertises "the first
routable address of the node", which is not deterministic once the node
is re-addressed, so it has to be stated either way. pod0 is a closed 10G
L2 between the three nodes; the management VLAN is about to become a
single 1G NIC shared with everything else management does, which is the
wrong link for quorum traffic. listenSubnets defaults to this, so peer
and client listeners move together.

This does move the etcd peer URLs off 10.2.40.x, so unlike the node IP
it is not a no-op: one node at a time, waiting for a healthy quorum
between each.

The API VIP stays on mgmt0; it is a Layer2VIPConfig and does not follow
the node IP.

Apply one node at a time. Each node re-IPs, so per the note in
k8s-1.yaml (#635) kube-apiserver keeps advertising the old address until
its static pod restarts: after each node, check that
`kubectl get endpoints kubernetes -n default` agrees with
`kubectl get nodes -o wide` and reboot the node if it does not. The
cilium agent reads the node IP only at startup, so it needs recreating
on each node as well.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

kubernetes/talos/patches/site-homelab.yaml

@@ -7,16 +7,32 @@
 # part of the config that could not be true anywhere but here.
 machine:
   kubelet:
-    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
-    # kubelet picks the first non-loopback IP, which could land on the v30iot
-    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
-    # must stay on mgmt LAN.
-    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
-    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
-    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
-    # provider prefix, which it then probes until the agent restarts (#789).
-    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
+    # Pins the node IP to pod0 (VLAN 48), which is what carries pod east-west.
+    # Cilium reaches a peer's pods at that peer's Kubernetes InternalIP and
+    # offers no way to point it elsewhere: --ipv4-node/--ipv6-node are accepted
+    # and then overwritten from the Node object, and directRoutingDevice only
+    # selects the NodePort/DSR device. So the node IP is the only control over
+    # which segment east-west uses, in both tunnel and native routing mode.
+    #
+    # Unpinned, kubelet takes the first non-loopback address, which could land
+    # on the v30iot NIC (ens19, 10.2.30.0/24).
+    #
+    # fd02:48::/64 is listed so the Node object carries an IPv6 InternalIP at
+    # all. Without one, Cilium detects the node's IPv6 itself and picks the
+    # SLAAC address from the dynamic provider prefix, then probes it until the
+    # agent restarts (#789).
     nodeIP:
       validSubnets:
-        - 10.2.40.0/24
-        - fd40::/64
+        - 10.2.48.0/24
+        - fd02:48::/64
+cluster:
+  etcd:
+    # Pins etcd to pod0 with the node IP. Unset, etcd advertises "the first
+    # routable address of the node", which is not deterministic across a
+    # re-address. listenSubnets defaults to this.
+    #
+    # pod0 is a closed 10G L2 between the three nodes. The management VLAN is
+    # about to become a single 1G NIC shared with everything else management
+    # does, which is the wrong link for quorum traffic.
+    advertisedSubnets:
+      - 10.2.48.0/24