fix(rook-ceph): keep the SAN pool on-link with the node addresses

The pool was 10.99.1.0/24, a separate subnet on the same segment, chosen to
avoid carving up a /29 with only three addresses left. That choice would have
broken the migration it exists for.

OSDs move one at a time, so the cluster runs mixed: an OSD already on the
attachment has to reach an OSD still using its node's host address on
10.99.0.x. There is no router on this segment — by design, it carries
Ceph-internal traffic and nothing else — so a pool outside the /29 is
unreachable from the un-migrated side. The two OSDs would mark each other
down, during precisely the window that is supposed to be the safe one.

Moves the pool into 10.99.0.0/29 with range_start/range_end reserving .4-.6,
leaving .1-.3 to the nodes. That is one address per OSD with nothing spare,
which is sufficient rather than merely tolerable: osdMaxUpdatesInParallel
bounds the migration to one OSD at a time, so allocations grow 1, 2, 3 and
never overlap. Headroom returns after the switch is confirmed and the node
addresses come out of the Talos configs — then the pool takes the whole /29.

Verified: the embedded CNI JSON parses with the intended range and bounds;
the NAD passes a server-side dry run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -29,13 +29,20 @@
 #   needed — unlike the v30iot attachments, there is no DHCP-supplied default
 #   route to confine, because there is no DHCP and no default route.
 #
-#   A separate subnet from the node addresses. The nodes hold 10.99.0.1-.3/29
-#   for as long as Ceph still uses host networking. OSDs never need to reach
-#   those — the cluster network is OSD-to-OSD — so this pool sits in its own
-#   range on the same segment instead of carving up a /29 that has exactly
-#   three addresses left. Once the provider switch is confirmed, the node
-#   addresses on VLAN 49 have no remaining purpose and can be removed from
-#   the Talos configs.
+#   The pool stays inside 10.99.0.0/29, on-link with the node addresses, and
+#   this is load-bearing during the migration rather than a detail. OSDs move
+#   one at a time, so the cluster runs mixed for a while: an OSD already on
+#   this attachment has to reach an OSD still using the node's host address.
+#   There is no router on this segment, so anything outside the /29 would be
+#   unreachable from the un-migrated side and the two would mark each other
+#   down — during precisely the window that is supposed to be safe.
+#
+#   range_start/range_end reserve .4-.6 and leave .1-.3 to the nodes. That is
+#   exactly one address per OSD with nothing spare, which is sufficient
+#   because osdMaxUpdatesInParallel bounds the migration to one OSD at a
+#   time: allocations grow 1, 2, 3 and never overlap. Once the switch is
+#   confirmed and the node addresses come out of the Talos configs, the pool
+#   can take the whole /29 and the spare returns.
 #
 # IPAM
 #
@@ -44,10 +51,6 @@
 # with no cluster-wide coordination, so two nodes would hand out the same
 # address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
 #
-# A /24 for three addresses is not extravagance — nothing else can ever live
-# on this segment to be crowded by it, and it removes address count as a
-# consideration when OSDs are added later.
-#
 # MTU 9000 matches the link. macvlan children inherit the master's MTU, but
 # it is stated so that a change to the parent does not silently halve
 # replication throughput without anything in the tree recording the
@@ -72,6 +75,8 @@ spec:
       "mtu": 9000,
       "ipam": {
         "type": "whereabouts",
-        "range": "10.99.1.0/24"
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.4",
+        "range_end": "10.99.0.6"
       }
     }