docs: runbook for converting a control-plane node to bare metal #584

2026-08-25T19:50:30Z

claude-bot commented

2026-08-25 19:50:30 +00:00

Written from doing jupiter on 2026-08-25, with saturn and neptune still to go.

The conversion itself was straightforward. Every failure came from state that outlived the VM, so the runbook is weighted towards those rather than the happy path — the ordering guidance in it is the actual deliverable.

The three that cost real time

The mon does not come back on its own. A rebuilt node gets a new MAC, so a new lease, and Ceph keeps the mon pinned to the old address in three places: the monmap, the rook-ceph-mon-endpoints ConfigMap, and --public-addr in the deployment. The mon can't bind an address the host no longer has → crashloops → the operator's next step wants to restart a healthy mon → Ceph refuses because that drops quorum → retries every 60s forever, never reaching the storage reconcile. Nothing behind it progresses, including Flux.

Order matters: fix the ConfigMap before the monmap and the deployment, or the operator faithfully recreates the mon from the stale entry (which is what happened). Rook then creates a mon with the next ID — jupiter's c came back as d — it does not repair the old one.

A wrong network config strands the node completely. Talos stops running default DHCP operators for the whole machine as soon as one new-style network document exists — the gate is presence-of-document, not success-of-resolution. Combined with a LinkAliasConfig selector that matches nothing being skipped silently (the type's own doc comment claims it errors; it doesn't), a small mistake became a node with no address on any interface, recoverable only from a console.

EPHEMERAL placement is one-shot. Honoured only when the volume does not yet exist. Applied later it's silently ignored and relocating needs a disk wipe plus install media.

Also recorded

Proxmox quorum, which starts mattering at the second conversion: two nodes with expected_votes: 2 means removing one leaves the last inquorate and /etc/pve read-only, blocking the final VM destroy. pvecm expected 1 fixes it and works while inquorate. Considered a QDevice and rejected — more moving parts installed onto something being deleted.
Secure Boot must be off; the Image Factory metal ISO is an unsigned UKI and MS-01 ships with it enabled.
Talos and Proxmox enumerate the same three NVMe drives differently — nvme0n1 on bare-metal jupiter is the Kingston system disk. Hence naming the OSD by /dev/disk/by-id.
The tofu destroy must happen while the host is still a PVE node, and AWS_* env vars are needed in addition to -backend-config for the S3 backend.
Known-stale things that will bite: bootstrap.sh can't generate a config (#581), manual Woodpecker pipelines are broken (#582), and Cilium silently drops VLAN-tagged traffic without bpf.vlanBypass — relevant since both mgmt0 and ens19 are VLANs on a converted node.

🤖 Generated with Claude Code

Written from doing jupiter on 2026-08-25, with saturn and neptune still to go. The conversion itself was straightforward. **Every failure came from state that outlived the VM**, so the runbook is weighted towards those rather than the happy path — the ordering guidance in it is the actual deliverable. ## The three that cost real time **The mon does not come back on its own.** A rebuilt node gets a new MAC, so a new lease, and Ceph keeps the mon pinned to the old address in three places: the monmap, the `rook-ceph-mon-endpoints` ConfigMap, and `--public-addr` in the deployment. The mon can't bind an address the host no longer has → crashloops → the operator's *next* step wants to restart a healthy mon → Ceph refuses because that drops quorum → retries every 60s forever, never reaching the storage reconcile. Nothing behind it progresses, including Flux. Order matters: fix the ConfigMap **before** the monmap and the deployment, or the operator faithfully recreates the mon from the stale entry (which is what happened). Rook then creates a mon with the *next* ID — jupiter's `c` came back as `d` — it does not repair the old one. **A wrong network config strands the node completely.** Talos stops running default DHCP operators *for the whole machine* as soon as one new-style network document exists — the gate is presence-of-document, not success-of-resolution. Combined with a `LinkAliasConfig` selector that matches nothing being skipped **silently** (the type's own doc comment claims it errors; it doesn't), a small mistake became a node with no address on any interface, recoverable only from a console. **EPHEMERAL placement is one-shot.** Honoured only when the volume does not yet exist. Applied later it's silently ignored and relocating needs a disk wipe plus install media. ## Also recorded - **Proxmox quorum**, which starts mattering at the *second* conversion: two nodes with `expected_votes: 2` means removing one leaves the last inquorate and `/etc/pve` read-only, blocking the final VM destroy. `pvecm expected 1` fixes it and works while inquorate. Considered a QDevice and rejected — more moving parts installed onto something being deleted. - **Secure Boot** must be off; the Image Factory metal ISO is an unsigned UKI and MS-01 ships with it enabled. - **Talos and Proxmox enumerate the same three NVMe drives differently** — `nvme0n1` on bare-metal jupiter is the Kingston *system* disk. Hence naming the OSD by `/dev/disk/by-id`. - The tofu destroy must happen while the host is still a PVE node, and `AWS_*` env vars are needed *in addition to* `-backend-config` for the S3 backend. - Known-stale things that will bite: `bootstrap.sh` can't generate a config (#581), manual Woodpecker pipelines are broken (#582), and Cilium silently drops VLAN-tagged traffic without `bpf.vlanBypass` — relevant since both `mgmt0` and `ens19` are VLANs on a converted node. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-25 19:50:30 +00:00

docs: runbook for converting a control-plane node to bare metal

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

7c9fa97423

Written from doing jupiter, with saturn and neptune still to go. The conversion
itself was straightforward; every failure came from state that outlived the VM,
so the runbook is weighted towards those rather than towards the happy path.

The three that cost real time:

A rebuilt node gets a new MAC and a new lease, and Ceph keeps its mon pinned to
the old address in the monmap, the mon-endpoints ConfigMap and the deployment's
--public-addr. The mon then cannot bind, the operator's next step wants to
restart a healthy mon, Ceph refuses because that would drop quorum, and it
retries every minute forever without reaching the storage reconcile. Fixing the
ConfigMap before the monmap and the deployment matters, or the operator
recreates the mon from the stale entry.

Talos stops running default DHCP for the whole machine as soon as one new-style
network document exists, and a LinkAliasConfig selector that matches nothing is
skipped silently. Together those turn a small config mistake into a node with no
address on any interface.

EPHEMERAL placement is honoured only on first provision, so it has to be right
in the first apply-config; afterwards it is ignored and relocating needs a wipe.

Also records the Proxmox quorum step that starts mattering at the second
conversion, the Secure Boot blocker, and that Talos and Proxmox enumerate the
same three NVMe drives in different orders — which is why the OSD is named by
by-id path rather than device name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-25 19:52:12 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 7c9fa974235db39913e92973f4f464543e26377c.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR adds a comprehensive runbook documenting the conversion of a control-plane node (jupiter) from Proxmox VM to bare metal, including Ceph OSD migration to a Samsung PM983. The runbook is well-structured, technically accurate, and grounded in hands-on experience with concrete warnings about failures that actually occurred.

🟢 Strengths

Accuracy: The instructions align with the actual cluster state. The cephcluster.yaml config (line 195-196) confirms the use of /dev/disk/by-id for the bare-metal node's OSD, and the three NVMe drives documented (Kingston system, Crucial EPHEMERAL, PM983 OSD) match the runbook exactly. The mon-address-pinning warnings in section 8 reflect known Ceph/Rook behavior patterns.

Practical gotchas: The three main warnings are grounded in real failures:

The mon address in three places (monmap, ConfigMap, deployment) is a concrete issue with Ceph rebuilds.
The network-config "all-or-nothing" behavior with DHCP operators (section 6) is accurate — Talos disables defaults as soon as one new-style network document exists, with no fallback.
The EPHEMERAL placement one-shot behavior (section 7) is correct and matches Talos documentation.

Structure and clarity: The runbook follows the project's established conventions (matching RUNBOOK-cold-rebuild.md). Checklist-formatted prerequisites, numbered steps with verification commands, and the "Known-stale things that will bite" conclusion are all helpful for operators.

Completeness: Each step includes both the action and how to verify it. Cross-references to related issues (#581, #582) provide context for known limitations. The warning about LinkAliasConfig selectors being silently skipped (section 6) is the kind of subtle detail that prevents hours of debugging.

🟡 Suggestions

Section 2a environment variables — The example shows AWS_ACCESS_KEY_ID=… with ellipsis, which is correct, but consider explicitly stating the credentials come from the S3 backend configuration. It could be clearer that these are in addition to the backend-config file, not a replacement for it. (The text does say this, but a note like "NB: these variables must be set in the shell, not in the backend config file" in the code block itself might help.)
Section 8 mon ordering — The critical ordering issue (ConfigMap → monmap → deployment) is stated once. Consider a callout box or more visual emphasis, since a reader who reads the three commands in isolation could still accidentally reverse them.
Section 9 by-id path — The runbook correctly warns against bare device names due to Talos enumeration differences from Proxmox. The note about talosctl get disk <dev> -o yaml to find symlinks is helpful, but explicitly showing the grep/jq path might be useful:
```
talosctl -n <node> get disk <dev> -o yaml | grep -A10 symlinks
```

🟢 Nits

Line 89: "Destroy the VM in OpenTofu — while the host is still a PVE node" — clear and important.
Line 151: Secure Boot warning is correctly emphasized. The technical detail about Proxmox using a Microsoft-signed shim vs. the unsigned UKI is accurate and necessary.
The reference to schematic a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64 matches the cold-rebuild runbook, good for consistency.

Tests

This is documentation, not code, so tests don't apply. However, the runbook's credibility is strengthened by:

It has been executed at least once (for jupiter), as evidenced by the current cluster state matching the documented configuration.
It references concrete failure modes that occurred during that execution.
The configuration files in the repo validate the technical claims made in the runbook.

A future PR validating the runbook on saturn and neptune would be valuable, but that is out of scope here.

Recommendation

Looks reasonable — no blocking issues. The runbook is accurate, grounded in real experience, and follows project conventions. The suggestions above are minor improvements to clarity; the content as-is is solid and actionable.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `7c9fa974235db39913e92973f4f464543e26377c`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR adds a comprehensive runbook documenting the conversion of a control-plane node (jupiter) from Proxmox VM to bare metal, including Ceph OSD migration to a Samsung PM983. The runbook is well-structured, technically accurate, and grounded in hands-on experience with concrete warnings about failures that actually occurred. ## 🟢 Strengths **Accuracy**: The instructions align with the actual cluster state. The cephcluster.yaml config (line 195-196) confirms the use of `/dev/disk/by-id` for the bare-metal node's OSD, and the three NVMe drives documented (Kingston system, Crucial EPHEMERAL, PM983 OSD) match the runbook exactly. The mon-address-pinning warnings in section 8 reflect known Ceph/Rook behavior patterns. **Practical gotchas**: The three main warnings are grounded in real failures: - The mon address in three places (monmap, ConfigMap, deployment) is a concrete issue with Ceph rebuilds. - The network-config "all-or-nothing" behavior with DHCP operators (section 6) is accurate — Talos disables defaults as soon as one new-style network document exists, with no fallback. - The EPHEMERAL placement one-shot behavior (section 7) is correct and matches Talos documentation. **Structure and clarity**: The runbook follows the project's established conventions (matching RUNBOOK-cold-rebuild.md). Checklist-formatted prerequisites, numbered steps with verification commands, and the "Known-stale things that will bite" conclusion are all helpful for operators. **Completeness**: Each step includes both the action and how to verify it. Cross-references to related issues (#581, #582) provide context for known limitations. The warning about `LinkAliasConfig` selectors being silently skipped (section 6) is the kind of subtle detail that prevents hours of debugging. ## 🟡 Suggestions 1. **Section 2a environment variables** — The example shows `AWS_ACCESS_KEY_ID=…` with ellipsis, which is correct, but consider explicitly stating the credentials come from the S3 backend configuration. It could be clearer that these are *in addition to* the backend-config file, not a replacement for it. (The text does say this, but a note like "NB: these variables must be set in the shell, not in the backend config file" in the code block itself might help.) 2. **Section 8 mon ordering** — The critical ordering issue (ConfigMap → monmap → deployment) is stated once. Consider a callout box or more visual emphasis, since a reader who reads the three commands in isolation could still accidentally reverse them. 3. **Section 9 by-id path** — The runbook correctly warns against bare device names due to Talos enumeration differences from Proxmox. The note about `talosctl get disk <dev> -o yaml` to find `symlinks` is helpful, but explicitly showing the grep/jq path might be useful: ```sh talosctl -n <node> get disk <dev> -o yaml | grep -A10 symlinks ``` ## 🟢 Nits - Line 89: "Destroy the VM in OpenTofu — while the host is still a PVE node" — clear and important. - Line 151: Secure Boot warning is correctly emphasized. The technical detail about Proxmox using a Microsoft-signed shim vs. the unsigned UKI is accurate and necessary. - The reference to schematic `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64` matches the cold-rebuild runbook, good for consistency. ## Tests This is documentation, not code, so tests don't apply. However, the runbook's credibility is strengthened by: - It has been executed at least once (for jupiter), as evidenced by the current cluster state matching the documented configuration. - It references concrete failure modes that occurred during that execution. - The configuration files in the repo validate the technical claims made in the runbook. A future PR validating the runbook on saturn and neptune would be valuable, but that is out of scope here. ## Recommendation Looks reasonable — no blocking issues. The runbook is accurate, grounded in real experience, and follows project conventions. The suggestions above are minor improvements to clarity; the content as-is is solid and actionable.

claude-bot referenced this pull request

2026-08-25 19:53:57 +00:00

chore: bring immich back up, keep cephfs-metadata degraded until the end #585

claude-bot force-pushed baremetal-conversion-runbook from 8523636af2 to 5e80a11f15

2026-08-25 19:56:45 +00:00

Compare

k.renner merged commit 39533bce25 into master

2026-08-25 19:57:03 +00:00

k.renner referenced this issue from a commit

2026-08-25 19:57:04 +00:00

Merge pull request 'docs: runbook for converting a control-plane node to bare metal' (#584) from baremetal-conversion-runbook into master

Sign in to join this conversation.