homelab/kubernetes/RUNBOOK-baremetal-conversion.md

Runbook — converting a control-plane node from Proxmox VM to bare metal

Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written after doing jupiter on 2026-08-25.

All three nodes are now on bare metal — talosctl get systeminformation reports Minisforum MS-01 hardware with distinct chassis serials for k8s-1, k8s-2 and k8s-3, with no hypervisor in between. Kept as the procedure for a rebuild, not as outstanding work.

Everything here that reads like an over-specific warning is something that actually happened. The conversion itself is straightforward — the failures were all in state that outlived the VM.

What the node ends up as

System disk	Kingston OM8PGP41024Q-A0, selected by serial
EPHEMERAL (/var)	the retired Crucial P3, selected by serial
Ceph OSD	Samsung PM983, selected by /dev/disk/by-id
Management	VLAN 40 tagged on X710 port 0
v30iot	VLAN 30 on the same port, named ens19
Ceph cluster net	X710 port 1, 10.99.0.x/29, MTU 9000

Before you start

• All three OSDs up, all mons in quorum, PGs active+clean. Do not start from a degraded cluster; the window itself removes one of three.
• Cluster fits on two nodes. Check kubectl describe node allocation on the two survivors. Take immich down (kubernetes/apps/immich, both Deployments to replicas: 0) — it frees ~1000m against a shortfall of roughly 200m and takes its image pulls off the remaining nodes.
• cephfs-metadata to size 2 / min_size 1 for the duration. At size 3 on two hosts every PG is undersized with min_size 2, so one further OSD loss blocks IO on every CephFS mount cluster-wide.
• Talos metal ISO on a USB stick. Schematic 0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9 (net.ifnames=1, intel-ucode, thunderbolt, i915; no qemu-guest-agent). Verify by comparing partition offsets and sizes against the ISO with fdisk -l — blkid labels lie after a rewrite.
• Physical access. Secure Boot must be turned off (see below), and that is the one step nothing can do remotely.

1. Evacuate the OSD

ceph osd out <id>

Wait until no PGs are backfilling or backfill_wait. Do not wait for ceph osd safe-to-destroy — pools at size 3 keep the OSD in their acting set because three copies cannot be placed on two hosts, so it never returns clean.

Then remove it. The node must already be out of cephcluster.yaml's storage.nodes list, or the operator rebuilds the OSD within one reconcile and starts backfilling onto the drive you are about to pull:

kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0
ceph osd purge <id> --yes-i-really-mean-it
kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>

2. Drain, and leave the cluster

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s

Single-replica workloads with minAvailable: 1 PDBs will refuse eviction — that is deliberate, they exist to surface the pod rather than let it be yanked. On this cluster that is influxdb-influxdb2-0, woodpecker-server and the CNPG primary. Move each one deliberately with kubectl delete pod; they reschedule onto the survivors and their RBD volumes follow.

talosctl -n <node> reset --graceful --wipe-mode system-disk
kubectl delete node <node>

--wipe-mode system-disk and not the default all: the OSD drive is being pulled anyway and wiping a terabyte of it is wasted time. --graceful leaves etcd cleanly. Delete the Node object — it carries the unschedulable flag from the drain, which the rebuilt node would otherwise inherit.

2a. Destroy the VM in OpenTofu — while the host is still a PVE node

Remove opentofu/vm_15N_k8s_N.tf, then apply. This must happen before the host leaves the Proxmox cluster: afterwards the provider cannot reach the node, the plan errors, and the only escape is tofu state rm, which is off the table.

Point proxmox_api_url at a host that will survive the conversion, not the one being converted.

Manual triggers are currently broken by the Woodpecker v3.18.0 regression (issue #582). Until that is resolved, run it locally:

AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=fsn1 \
  tofu -chdir=opentofu init -backend-config=<file>
… plan -target=module.k8s_N -var-file=<file> -out=<plan>
… apply <plan>

The AWS_* environment variables are required in addition to -backend-config, or the S3 backend fails HeadObject with 403 even with credentials that can read the object fine via the aws CLI.

2b. Proxmox quorum — matters from the second conversion onward

Two nodes remain after jupiter, with expected_votes: 2 and quorum 2. Removing one leaves the last node inquorate and /etc/pve read-only, which blocks the final VM destroy. On the surviving node:

pvecm expected 1

This is the official recovery. Proxmox documents it in the storage replication chapter (/usr/share/pve-docs/chapter-pvesr.html), verbatim:

If you have no quorum, we strongly advise to fix this first and make the node operable again. Only if this is not possible at the moment, you may use the following command to enforce quorum on the current node: # pvecm expected 1

The caveat there is about restoring the missing node instead. Here the missing node is deliberately gone for good, so enforcing quorum is the correct branch rather than a workaround.

Run it only once the node is actually alone — attempting it while both nodes are up fails with CS_ERR_INVALID_PARAM, since corosync will not set expected votes below the members it can currently see. Which is exactly why it becomes valid when one remains.

Runtime only; it does not survive a reboot, so re-run it if the survivor restarts before its own conversion.

3. Hardware

Power the host off, then:

1. Kingston boot drive out of slot 1, into slot 2.
2. Crucial P3 out of slot 2 — it becomes EPHEMERAL, so put it in slot 3 rather than setting it aside.
3. Slot 1 switch to U.2. Set it before the drive goes in; there is nothing to record afterwards, the installed drive type implies the position.
4. PM983 onto the included 7mm M.2-to-U.2 adapter, into slot 1.

Boot the host's existing PVE once before touching Talos and check the drive:

nvme smart-log /dev/nvmeXn1 | grep -iE 'percentage_used|media_errors|critical'
nvme id-ns /dev/nvmeXn1 -H | grep -i 'in use'      # want 512b data, 0 metadata
wipefs /dev/nvmeYn1                                 # Crucial: clear the old bluestore sig
ip link set <ceph-port> up && ethtool <ceph-port> | grep -i 'link detected'

This is the last revert point. A bad drive here means the Crucial goes back and the VM is recreated from tofu with nothing lost. Once Talos writes to the Kingston, PVE on that host is gone.

Relocating the boot drive is safe: rpool is referenced by NVMe EUI and the UEFI entries by GPT PARTUUID, neither of which encodes a slot.

4. Secure Boot

MS-01 ships with Secure Boot on, and Proxmox satisfies it with a Microsoft-signed shim. The Image Factory metal-amd64.iso is an unsigned UKI and the firmware rejects it outright. Turn Secure Boot off in the BIOS.

A signed variant exists per schematic, but it is signed with SideroLabs' key which firmware does not trust by default, and the installed system would have to use the secureboot installer image too. That is a separate project.

5. Install

Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts that lands on 192.168.8.x, which is routable from the workstation. It DHCPs on every link, so one machine answers on several addresses at once — they are the same host, not several hosts. get disks on each confirms it.

Maintenance mode does serve talosctl get. --insecure is a flag on the subcommand rather than a global, so it must come after get — placed before it, cobra resolves to the legacy talosctl disks command and reports unknown flag: --insecure:

talosctl -n <maintenance-ip> get disks --insecure
talosctl -n <maintenance-ip> get links --insecure

Verify serials and MACs here, on the machine itself, against the config about to be applied.

Build the config from the live cluster's secrets, not a fresh talosctl gen config, or the node forms a new cluster instead of joining:

talosctl -n <surviving-node> get mc v1alpha1 -o jsonpath='{.spec}' > cp.yaml
talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
talosctl gen config homelab https://k8s.renner.uno:6443 \
  --with-secrets secrets.yaml --output-types controlplane \
  --config-patch @talos/patches/cluster.yaml \
  --config-patch @talos/patches/containerd-layer-retention.yaml \
  --config-patch @talos/patches/k8s-N.yaml -o install.yaml

Confirm cluster.id matches the running cluster before applying.

Strip the generated HostnameConfig document — it collides with machine.network.hostname from the per-node patch. bootstrap.sh does the same.

There is no CNI patch to pass any more. Talos supplies only cni: none and proxy: disabled; Cilium is installed from its HelmRelease values, which is Step 2b of RUNBOOK-cold-rebuild.md. A node converted into an existing cluster gets its CNI from the Cilium DaemonSet already running there, so nothing extra is needed here.

Validate:

talosctl validate --config install.yaml --mode metal

Install minimal, complete on the running node

Do not apply the full per-node config in maintenance mode. A document that references an interface which does not exist yet hangs the install: the node sits at STAGE: Installing indefinitely with KUBELET n/a, APISERVER n/a, and the console scrolling

k8s.NodeApplyController: error getting node: nodes "k8s-N" not found

That message is a symptom of an install that never finished, not a Kubernetes or networking fault — the node holds a DHCP address and is reachable throughout. On saturn the offenders were the ens19 VLAN document and the sysctl naming it, net.ipv6.conf.ens19.disable_ipv6. Three cold installs hung on it; the identical config applied cleanly once the node was up. Wiping disks makes no difference — it is not leftover state.

Stage 1 — the smallest config that installs and joins, plus EPHEMERAL, which is one-shot (section 7) and so cannot be deferred to stage 2:

machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
LinkConfig    <uplink>
VLANConfig    mgmt0
DHCPv4Config  mgmt0
VolumeConfig  EPHEMERAL

talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml

Stage 2 — once the node is Ready and talosctl etcd members lists it, apply the full config to the running node. --mode=try reverts on its own if the network config is wrong, so it cannot strand the node; re-apply with --mode=auto to persist.

talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml

Both applied without a reboot on saturn. Confirm ens19, mgmt0 and the Ceph link are up afterwards with talosctl get links and get addresses.

6. The network config will strand the node if it is wrong

Talos stops running default DHCP operators for the entire machine as soon as a single new-style network document exists — the gate is presence of a document, not success of resolution. If the configured path fails to materialise there is no fallback on any interface, and recovery needs a console and install media.

Concretely: do not use LinkAliasConfig as a VLANConfig parent without verifying it. Aliases are supported (confirmed in v1.12.6 source and by a maintainer), but a selector matching zero links is skipped silently, and permanent_addr comes from an ethtool ioctl with no fallback — if the driver does not answer, the match is against an empty string and everything downstream that references the alias does nothing.

Address links by predictable kernel name. To introduce aliases later, do it on a node that is already up, with talosctl apply-config --mode=try --timeout=5m, which rolls back on its own, and check talosctl get linkaliasspec.

Interface names derive from PCI topology and are stable — but not immune. Adding a third NVMe to jupiter renumbered the buses and renamed the unused second I226 from enp89s0 to enp90s0.

7. EPHEMERAL placement is one-shot

apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
  diskSelector:
    match: disk.serial == '<crucial serial>'
  minSize: 50GiB
  grow: true

Talos applies volume configuration only when the volume does not yet exist. Applied afterwards it is silently ignored, and relocating EPHEMERAL then needs a disk wipe and a fresh boot from install media. It must be in the first apply-config.

Match by serial. disk.transport == 'nvme' && !system_disk matches the PM983 as well, and losing that coin flip formats the OSD drive.

Verify afterwards:

talosctl -n <node> get volumestatus EPHEMERAL -o yaml | grep -E 'location|phase'
talosctl -n <node> get disks

8. The mon will not come back on its own

This is the step that cost the most time on jupiter. The rebuilt node has a new MAC and therefore a new DHCP lease, but Ceph has its mon pinned to the old address in three places. The mon cannot bind an address the host no longer has, so it crashloops; the operator then tries to restart a healthy mon as its next step, Ceph refuses because that would drop quorum, and it retries every 60 seconds forever without ever reaching the storage reconcile. Nothing else progresses behind it.

Fix in this order — the order matters, because doing it the other way round has the operator faithfully recreate the mon from the stale ConfigMap:

# 1. ConfigMap first
kubectl -n rook-ceph edit cm rook-ceph-mon-endpoints
#    remove the dead mon from `data` and from `mapping`

# 2. then the monmap
ceph mon remove <id>

# 3. then the deployment — check it really carries the old address
kubectl -n rook-ceph get deploy rook-ceph-mon-<id> -o yaml | grep public-addr
kubectl -n rook-ceph delete deploy rook-ceph-mon-<id>

Rook then creates a new mon with the next free ID (jupiter's c came back as d), on the correct address. It does not repair the old one.

Note mons use host ports under network.provider: host, so a replacement mon cannot schedule while the broken one's deployment still holds 3300/6789 — the canary sits Pending with "didn't have free ports" until the old deployment is deleted.

9. Re-add the OSD

Add the node back to cephcluster.yaml under storage.nodes, naming the PM983 by /dev/disk/by-id:

- name: k8s-N
  devices:
    - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_<serial>

Not a bare device name. Talos enumerates these drives in a different order than Proxmox did — on jupiter nvme0n1 is the Kingston system disk, and the PM983 is nvme2n1. Rook would probably decline a partitioned system disk, but that is luck rather than design.

Get the path from talosctl -n <node> get disk <dev> -o yaml, under symlinks.

10. Verify and restore

kubectl get nodes                                   # Ready
talosctl -n <node> etcd members                     # 3 members
ceph status                                         # 3 mons in quorum, 3 osds
ceph osd tree                                       # OSD under the right zone/host
ceph device ls                                      # PM983 backing the new OSD

Then undo the window changes: cephfs-metadata back to size 3, immich back to 1 replica, and ceph config set osd osd_mclock_profile balanced if it was raised to high_recovery_ops to speed the evacuation.

The retired Crucial leaves a stale entry in ceph device ls. Remove it with ceph config-key rm device/<devid>; the mgr's devicehealth view may lag.

Long-lived TCP sessions to devices outside the cluster die silently

Check this before declaring the conversion done. Anything holding a persistent connection to a device on another VLAN — the ESPHome integration is the case that bit us — keeps a half-open socket when the node underneath it moves. The peer drops its side; the pod's side never learns, because no RST arrives and no keepalive fires.

The symptom is that there is no symptom. Home Assistant logged nothing, retried nothing, and reported the devices as connected. The devices themselves showed "not connected". Hours went into reading logs that were empty by construction.

/proc/net/tcp in the pod settles it in one command — 01 is ESTABLISHED, 02 is SYN_SENT, and the port is hex:

kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \
  sh -c 'grep ":17A5" /proc/net/tcp'      # 17A5 = 6053, the ESPHome API

Seven 01 entries to devices insisting they were disconnected is the tell. A 02 alongside them is a genuinely dead device, not a stale socket — that one Home Assistant was retrying and did log.

Restarting the consuming pod clears it. Reloading just the integration from the UI does the same with less disruption. Neither is discoverable from logs, which is why it belongs here.

Known-stale things that will bite

• bootstrap.sh cannot generate a config at all right now (issue #581), so a cold rebuild is broken independently of this work.

• Manual Woodpecker pipelines are broken (issue #582), so anything gated on PIPELINE == "..." must be run another way.

• Cilium's eBPF datapath drops VLAN-tagged traffic unless bpf.vlanBypass is set. This happened — every v30iot macvlan workload on a converted node stayed in sandbox creation until vlanBypass: [30] was added (issue #592).

  Only ens19 is affected, and the reason is worth knowing: Cilium attaches to devices that carry an address. cilium-dbg status --verbose lists mgmt0 among its devices but not ens19, because ens19 is deliberately address-less. Cilium therefore owns mgmt0 and sees VLAN 40 frames already untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter on the shared parent uplink, still tagged. Any future address-less VLAN carrying pod traffic will need the same treatment.

  The symptom does not look like a VLAN problem. A tc-layer drop reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports no buffer space available and network is down rather than a DHCP timeout, and the pod sits in ContainerCreating behind plugin type="macvlan" failed (add): error calling DHCP.Allocate: no more tries. Confirm it with cilium_drop_count_total{reason=~"VLAN.*"} per node rather than by reading the CNI logs, and check node_network_receive_packets_total{device="ens19"} — it reads 0 on an affected node.