|
|
|
@@ -1,103 +1,194 @@
|
|
|
|
|
# Proposal — separate the four traffic classes onto their own VLANs
|
|
|
|
|
|
|
|
|
|
Status: open, issue #576. Two of the four classes are separated. Pod east-west
|
|
|
|
|
cannot leave the node IP under Cilium. Both ways of moving it within Cilium
|
|
|
|
|
(options A and B below) are rejected by the owner, and replacing the CNI
|
|
|
|
|
(option C) is shelved.
|
|
|
|
|
Status: draft. The question this started with is **answered** — from the Cilium
|
|
|
|
|
source and flag reference, not from a test — and the answer changes the shape
|
|
|
|
|
of the proposal rather than confirming it.
|
|
|
|
|
|
|
|
|
|
## Target
|
|
|
|
|
Supersedes the framing in issue #576, which described this as "give node
|
|
|
|
|
management its own NIC, and Ceph public its own VLAN". That was two of four
|
|
|
|
|
classes and one of them turned out already solved.
|
|
|
|
|
|
|
|
|
|
Set by the owner on 2026-09-08:
|
|
|
|
|
## The four classes
|
|
|
|
|
|
|
|
|
|
| class | link |
|
|
|
|
|
|---|---|
|
|
|
|
|
| Ceph replication | X710 p1, VLAN 49 |
|
|
|
|
|
| pod-pod | X710 trunk, a VLAN of its own |
|
|
|
|
|
| world-pod | X710 trunk, a VLAN of its own |
|
|
|
|
|
| node management | 1 Gb onboard NIC |
|
|
|
|
|
|
|
|
|
|
Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
|
|
|
|
|
on the 10G switch.
|
|
|
|
|
|
|
|
|
|
## Where the classes are
|
|
|
|
|
|
|
|
|
|
Measured 2026-09-15.
|
|
|
|
|
|
|
|
|
|
| class | today | state |
|
|
|
|
|
| class | what it is | where it goes today |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
|
|
|
|
|
| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
|
|
|
|
|
| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
|
|
|
|
|
| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
|
|
|
|
|
| management | `mgmt0`, VLAN 40 | not started |
|
|
|
|
|
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
|
|
|
|
|
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
|
|
|
|
|
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
|
|
|
|
|
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
|
|
|
|
|
|
|
|
|
|
`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
|
|
|
|
|
#833). `pod0` holds addresses and carries no traffic.
|
|
|
|
|
Three of four share one interface. That is not a design decision — until the
|
|
|
|
|
bare-metal conversion there was one NIC to work with.
|
|
|
|
|
|
|
|
|
|
## Why pod-pod is blocked
|
|
|
|
|
Ceph client I/O living on the pod network is correct rather than a leftover:
|
|
|
|
|
that traffic *is* cluster-internal. Ceph is done.
|
|
|
|
|
|
|
|
|
|
Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
|
|
|
|
|
next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:
|
|
|
|
|
## Principle
|
|
|
|
|
|
|
|
|
|
- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
|
|
|
|
|
fed from `newNode.GetNodeIP(false)`.
|
|
|
|
|
- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
|
|
|
|
|
- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
|
|
|
|
|
`initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
|
|
|
|
|
Node object.
|
|
|
|
|
- `--direct-routing-device` selects a device for the BPF NodePort path, not a
|
|
|
|
|
next-hop.
|
|
|
|
|
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
|
|
|
|
|
separate, changeable decision.**
|
|
|
|
|
|
|
|
|
|
Tested on the cluster, none of these moved the next-hop:
|
|
|
|
|
Defining classes by VLAN means the separation survives recabling, port failures
|
|
|
|
|
and capacity changes, and it does not require one port per class. Defining them
|
|
|
|
|
by interface would freeze a capacity decision into the architecture.
|
|
|
|
|
|
|
|
|
|
| attempt | result |
|
|
|
|
|
|---|---|
|
|
|
|
|
| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
|
|
|
|
|
| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
|
|
|
|
|
| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |
|
|
|
|
|
## The finding that shapes everything else
|
|
|
|
|
|
|
|
|
|
The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
|
|
|
|
|
`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
|
|
|
|
|
Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
|
|
|
|
|
cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
|
|
|
|
|
has no effect on pod-pod routing.
|
|
|
|
|
The original open question was whether `nodePort.directRoutingDevice` moves the
|
|
|
|
|
next-hop of Cilium's direct routes, or only selects an interface. It is
|
|
|
|
|
answerable from the source, and the answer is: **only the interface.**
|
|
|
|
|
|
|
|
|
|
## Options for pod-pod
|
|
|
|
|
`installDirectRoute` takes the next-hop as a parameter:
|
|
|
|
|
|
|
|
|
|
| | option | status |
|
|
|
|
|
```go
|
|
|
|
|
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
|
|
|
|
|
skipUnreachable bool) (routeSpec *netlink.Route, err error)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
|
|
|
|
|
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
|
|
|
|
|
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
|
|
|
|
|
|
|
|
|
|
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
|
|
|
|
|
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
|
|
|
|
|
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
|
|
|
|
|
cluster there is no path by which Cilium's node IP differs from the kubelet's
|
|
|
|
|
`InternalIP`.
|
|
|
|
|
|
|
|
|
|
`--direct-routing-device` is documented as "Device name used to connect nodes in
|
|
|
|
|
direct routing mode" — a device, not an address.
|
|
|
|
|
|
|
|
|
|
**So cluster-internal traffic is welded to the node IP, and the node IP is the
|
|
|
|
|
kubelet InternalIP.** Two consequences:
|
|
|
|
|
|
|
|
|
|
- The class cannot be moved off whatever VLAN the node IP lives on.
|
|
|
|
|
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
|
|
|
|
|
from the node IP; native mode uses it as the direct-route next-hop. Native
|
|
|
|
|
routing buys nothing here, so the disruptive, non-rolling step this proposal
|
|
|
|
|
previously contained is simply gone.
|
|
|
|
|
|
|
|
|
|
## Consequence: draw the line the other way
|
|
|
|
|
|
|
|
|
|
If cluster-internal cannot leave the node IP, then separation has to come from
|
|
|
|
|
moving *management services* off it — and each one has to be checked
|
|
|
|
|
individually, because they do not share a mechanism.
|
|
|
|
|
|
|
|
|
|
| management service | can it leave the node IP? | lever |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
|
|
|
|
|
| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
|
|
|
|
|
| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |
|
|
|
|
|
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
|
|
|
|
|
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
|
|
|
|
|
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
|
|
|
|
|
| kubelet | **no** | it *is* the node IP, by definition |
|
|
|
|
|
|
|
|
|
|
A per-workload Multus attachment detaches a single workload from the node IP,
|
|
|
|
|
the way Ceph replication is detached. It does not cover east-west traffic in
|
|
|
|
|
So four VLANs are reachable, but the boundary does not fall where the class
|
|
|
|
|
names suggest: **kubelet stays with cluster-internal.** That is the honest
|
|
|
|
|
result, and it is worth stating plainly rather than filing kubelet under
|
|
|
|
|
management and hoping.
|
|
|
|
|
|
|
|
|
|
The remaining classes keep the levers established earlier:
|
|
|
|
|
|
|
|
|
|
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
|
|
|
|
|
For eBGP the session source is the next-hop, so pinning the session pins
|
|
|
|
|
where return traffic arrives. The session to the UDM has been running for
|
|
|
|
|
months.
|
|
|
|
|
- **cluster-external egress** — eBPF masquerading follows the output device, so
|
|
|
|
|
this is a default-route decision plus `enable-masquerade-to-route-source`
|
|
|
|
|
(today `false`).
|
|
|
|
|
- **Ceph replication** — Multus macvlan on VLAN 49, done.
|
|
|
|
|
|
|
|
|
|
## The general escape hatch
|
|
|
|
|
|
|
|
|
|
Ceph is separated because its OSDs got a second interface and stopped using the
|
|
|
|
|
pod network for replication. That is the *only* mechanism that fully detaches a
|
|
|
|
|
workload from the node IP, and it generalises: anything that must not ride the
|
|
|
|
|
node IP gets its own `NetworkAttachmentDefinition`.
|
|
|
|
|
|
|
|
|
|
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
|
|
|
|
|
change. Worth it for Ceph replication. Not worth it for east-west traffic in
|
|
|
|
|
general.
|
|
|
|
|
|
|
|
|
|
## Levers per management service
|
|
|
|
|
## Recommended mapping
|
|
|
|
|
|
|
|
|
|
These hold regardless of which option is chosen:
|
|
|
|
|
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
|
|
|
|
|
|
|
|
|
|
| service | can leave the node IP? | lever |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
|
|
|
|
|
| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
|
|
|
|
|
| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
|
|
|
|
|
| kubelet | no | it is the node IP |
|
|
|
|
|
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
|
|
|
|
|
and Talos API. Cleanest match to the class names, but it changes every node's
|
|
|
|
|
identity, which is the one address the whole cluster is administered through.
|
|
|
|
|
|
|
|
|
|
## Remaining, independent of pod-pod
|
|
|
|
|
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
|
|
|
|
|
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
|
|
|
|
|
Identical end state, and no node ever changes its InternalIP.
|
|
|
|
|
|
|
|
|
|
- **pod→world egress**: default route and masquerade source onto `ingress0`,
|
|
|
|
|
via route metrics, `devices` and `enable-masquerade-to-route-source`. This
|
|
|
|
|
moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
|
|
|
|
|
- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
|
|
|
|
|
removed once ingress on VLAN 47 is settled.
|
|
|
|
|
Under B:
|
|
|
|
|
|
|
|
|
|
## Sources
|
|
|
|
|
| VLAN | carries |
|
|
|
|
|
|---|---|
|
|
|
|
|
| 49 | Ceph replication |
|
|
|
|
|
| 40 (relabelled *cluster*) | pod east-west, kubelet |
|
|
|
|
|
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
|
|
|
|
|
| new *external* | LoadBalancer ingress, egress default route |
|
|
|
|
|
|
|
|
|
|
- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
|
|
|
|
|
`--direct-routing-device`.
|
|
|
|
|
- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
|
|
|
|
|
- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
|
|
|
|
|
- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
|
|
|
|
|
`CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
|
|
|
|
|
The property the user asked for — management carries only management traffic —
|
|
|
|
|
holds under B. The label moves; the property is what matters.
|
|
|
|
|
|
|
|
|
|
## Migration shape
|
|
|
|
|
|
|
|
|
|
Every step is rolling and reversible. The routing-mode change is gone.
|
|
|
|
|
|
|
|
|
|
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
|
|
|
|
|
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
|
|
|
|
|
Still unused; this only proves the L2 works.
|
|
|
|
|
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
|
|
|
|
|
waiting for a healthy quorum between each.
|
|
|
|
|
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
|
|
|
|
|
`k8s.renner.uno` follows.
|
|
|
|
|
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
|
|
|
|
|
Reversible by reverting the override; failure is visible immediately in
|
|
|
|
|
`cilium-dbg shell -- bgp/peers`.
|
|
|
|
|
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
|
|
|
|
|
|
|
|
|
|
Step 3 is the one that deserves care — etcd is the only step where a mistake
|
|
|
|
|
costs quorum rather than connectivity.
|
|
|
|
|
|
|
|
|
|
## What this does not buy
|
|
|
|
|
|
|
|
|
|
Management currently shares a 10 Gb link and is not contended. Nothing observed
|
|
|
|
|
today is slow because of this. The case for the split is that traffic classes
|
|
|
|
|
with different failure modes and security postures should not be
|
|
|
|
|
indistinguishable — not that something is broken.
|
|
|
|
|
|
|
|
|
|
## Sources and measurements
|
|
|
|
|
|
|
|
|
|
Source, Cilium v1.20.1 (the running version):
|
|
|
|
|
|
|
|
|
|
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
|
|
|
|
|
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
|
|
|
|
|
a next-hop that is not directly reachable.
|
|
|
|
|
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
|
|
|
|
|
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
|
|
|
|
|
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
|
|
|
|
|
Kubernetes Node.
|
|
|
|
|
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
|
|
|
|
|
to connect nodes in direct routing mode".
|
|
|
|
|
|
|
|
|
|
Talos:
|
|
|
|
|
|
|
|
|
|
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
|
|
|
|
|
defaults to it. kube-apiserver and Talos API listen addresses are not
|
|
|
|
|
covered there.
|
|
|
|
|
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
|
|
|
|
|
`10.2.40.60` with `link: mgmt0`.
|
|
|
|
|
|
|
|
|
|
Measured on the cluster:
|
|
|
|
|
|
|
|
|
|
- `apid` answered on a node address that is not the InternalIP, so the Talos
|
|
|
|
|
API is not bound to one address.
|
|
|
|
|
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
|
|
|
|
|
InternalIP had changed since the last session, which is why nothing here
|
|
|
|
|
names an address as fixed.
|
|
|
|
|
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
|
|
|
|
|
unset and no direct-routing device configured.
|
|
|
|
|
|
|
|
|
|
Not verified: whether the kube-apiserver's `--advertise-address` follows the
|
|
|
|
|
node IP independently of the VIP. It does not affect the recommendation, since
|
|
|
|
|
clients reach the API through the VIP either way.
|
|
|
|
|
claude-botandClaude Opus 5
k.renner
security-bot