feat(k8s-1): run Talos on bare-metal jupiter and swap the OSD to U.2 #566

Merged

k.renner merged 1 commits from jupiter-baremetal-u2 into master 2026-08-25 14:51:53 +00:00

Conversation 1 Commits 1 Files Changed 6

+150 -111

claude-bot commented 2026-08-25 14:46:30 +00:00

Member

Copy Link

Copy Source

jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the MS-01's only U.2-capable slot means relocating the boot drive anyway. Doing the bare-metal conversion in the same window avoids opening the machine twice and throws away no work — the osd-ssd PCI mapping a Proxmox-side swap would need re-pointed simply ceases to exist.

Note this departs from the recorded single-end-cutover policy for the Proxmox retirement: the cluster runs mixed (k8s-1 bare metal, k8s-2/k8s-3 still VMs) until saturn and neptune follow.

Changes

• talos/patches/k8s-1.yaml — rewritten for bare metal. Uses the v1.12 multi-document network config (LinkAliasConfig/LinkConfig/VLANConfig/DHCPv4Config/Layer2VIPConfig) rather than the deprecated machine.network.interfaces, because the docs do not define behaviour when one link is configured by both schemas. Verified with talosctl validate --mode metal against v1.12.6.
• v30iot — the VLAN 30 link is named ens19. VLANConfig.name is free-form, so the four macvlan NADs that hard-code master: ens19 keep working on both VM and bare-metal nodes. No workload pinning needed, and no NAD changes.
• NIC selection by permanent MAC — both X710 ports bind i40e on bare metal, so the driver selector k8s-2/k8s-3 use would be ambiguous here.
• Install disk by serial (50026B7382B48B86, the Kingston) — with passthrough gone Talos sees both NVMe drives, and the installer must never land on the OSD drive.
• opentofu/vm_150_k8s_1.tf deleted — must be applied while jupiter still runs PVE, or the resource is orphaned in state.
• deploy-infrastructure.yml — jupiter dropped from the Talos image seeding loop; it is no longer a Proxmox node and the download-wait would burn 60x10s per run.
• cephfs-metadata → size 2 / min_size 1 — temporary. At size 3 on two remaining hosts every PG is undersized with min_size 2, so a single further OSD loss blocks IO on every CephFS mount cluster-wide. Must be reverted to size 3 once k8s-1 rejoins.
• RUNBOOK-cold-rebuild — records the bare-metal image, which is a different Image Factory schematic (a21cec8f…): drops qemu-guest-agent, adds intel-ucode, adds thunderbolt (inert until machine.kernel.modules requests it).

Not in this PR

cephcluster.yaml's QLC/wear rationale block still describes a uniform Crucial P3 Plus fleet. Left alone until the PM983 is in and its real SMART figures are known.

🤖 Generated with Claude Code

jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the MS-01's only U.2-capable slot means relocating the boot drive anyway. Doing the bare-metal conversion in the same window avoids opening the machine twice and throws away no work — the `osd-ssd` PCI mapping a Proxmox-side swap would need re-pointed simply ceases to exist. Note this departs from the recorded single-end-cutover policy for the Proxmox retirement: the cluster runs mixed (k8s-1 bare metal, k8s-2/k8s-3 still VMs) until saturn and neptune follow. ## Changes - **`talos/patches/k8s-1.yaml`** — rewritten for bare metal. Uses the v1.12 multi-document network config (`LinkAliasConfig`/`LinkConfig`/`VLANConfig`/`DHCPv4Config`/`Layer2VIPConfig`) rather than the deprecated `machine.network.interfaces`, because the docs do not define behaviour when one link is configured by both schemas. Verified with `talosctl validate --mode metal` against v1.12.6. - **v30iot** — the VLAN 30 link is named `ens19`. `VLANConfig.name` is free-form, so the four macvlan NADs that hard-code `master: ens19` keep working on both VM and bare-metal nodes. No workload pinning needed, and no NAD changes. - **NIC selection by permanent MAC** — both X710 ports bind `i40e` on bare metal, so the driver selector k8s-2/k8s-3 use would be ambiguous here. - **Install disk by serial** (`50026B7382B48B86`, the Kingston) — with passthrough gone Talos sees both NVMe drives, and the installer must never land on the OSD drive. - **`opentofu/vm_150_k8s_1.tf` deleted** — must be applied while jupiter still runs PVE, or the resource is orphaned in state. - **`deploy-infrastructure.yml`** — jupiter dropped from the Talos image seeding loop; it is no longer a Proxmox node and the download-wait would burn 60x10s per run. - **`cephfs-metadata` → size 2 / min_size 1** — temporary. At size 3 on two remaining hosts every PG is undersized with min_size 2, so a single further OSD loss blocks IO on every CephFS mount cluster-wide. **Must be reverted to size 3 once k8s-1 rejoins.** - **RUNBOOK-cold-rebuild** — records the bare-metal image, which is a different Image Factory schematic (`a21cec8f…`): drops `qemu-guest-agent`, adds `intel-ucode`, adds `thunderbolt` (inert until `machine.kernel.modules` requests it). ## Not in this PR `cephcluster.yaml`'s QLC/wear rationale block still describes a uniform Crucial P3 Plus fleet. Left alone until the PM983 is in and its real SMART figures are known. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-25 14:46:31 +00:00

feat(k8s-1): run Talos on bare-metal jupiter and swap the OSD to U.2 ... 6380f02ca4

jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the
MS-01's only U.2-capable slot means moving the boot drive anyway. Doing the
bare-metal conversion in the same window avoids opening the machine twice and
throws away no work: the osd-ssd PCI mapping that a Proxmox-side swap would
need to be re-pointed at ceases to exist.

Networking uses the v1.12 multi-document config rather than the deprecated
machine.network.interfaces. The docs do not define what happens when one link
is configured by both schemas, so k8s-1 keeps every link in one of them. This
also buys VLANConfig.name, which is free-form: naming the VLAN 30 link ens19
keeps the four v30iot macvlan NADs working unchanged across VM and bare-metal
nodes, so the cluster can stay mixed without pinning those workloads.

Physical NICs are matched by permanent MAC, not driver — both X710 ports bind
i40e here, unlike the single passed-through port k8s-2/k8s-3 see.

cephfs-metadata drops to size 2 / min_size 1 for the duration. At size 3 on
two remaining hosts every PG sits undersized with min_size 2, so one further
OSD loss would block IO on every CephFS mount cluster-wide. This must go back
to size 3 once k8s-1 rejoins.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 3e57ed5c2d into master 2026-08-25 14:51:53 +00:00

k.renner referenced this issue from a commit 2026-08-25 14:51:54 +00:00

Merge pull request 'feat(k8s-1): run Talos on bare-metal jupiter and swap the OSD to U.2' (#566) from jupiter-baremetal-u2 into master

review-bot requested changes 2026-08-25 14:53:01 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 6380f02ca4941c15953330f4aaf4d10c5fb71a97.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

This PR converts k8s-1 (jupiter) from a Proxmox VM to bare-metal Talos, rewrites its Talos network config onto the v1.12 multi-document schema, moves the Ceph OSD to a native NVMe (U.2/PM983) drive, drops the now-orphaned opentofu/vm_150_k8s_1.tf, and temporarily relaxes cephfs-metadata to size 2/min_size 1 while the cluster runs on two remaining Ceph hosts. The reasoning in the PR body and inline comments is unusually thorough and each documented trade-off (VIP handling, NIC selection, install-disk-by-serial, the OSD-vs-system-disk enumeration risk, the metadata-pool availability trade) checks out against the code. The gap is scope: several other files in the repo still hard-wire jupiter as a Proxmox node and are not touched here, which will break once jupiter stops running PVE.

🔴 Blocking

• ansible/inventory/group_vars/all/vars.yml:2 — pve_api_url: "https://jupiter.renner.uno:8006" is unchanged. It's consumed by ansible/roles/vm-update/tasks/main.yml:66 and :82 to trigger and poll vzdump backups via {{ pve_api_url }}/api2/json/nodes/{{ proxmox_node }}/vzdump for any Proxmox node. Since a Proxmox cluster's API is reachable from any member and this URL is the sole entry point configured, once jupiter no longer runs PVE this breaks backup-trigger automation for the entire remaining fleet (saturn/neptune, i.e. k8s-2/k8s-3 and any other guests) — not just for jupiter's own (now-removed) VM.
• ansible/inventory/api.proxmox.yaml:5 — the community.proxmox.proxmox dynamic-inventory plugin is pointed at https://jupiter.renner.uno:8006 as its sole url. Same issue: this is the only configured entry point for cluster-wide dynamic inventory discovery, and it stops answering once jupiter drops PVE.
• kubernetes/apps/pve/endpointslice.yaml:19-23 — jupiter is still listed as an endpoint (10.2.40.201:8006, ready: true) behind the pve Service. helmrelease-prometheus.yaml's pve (cluster-mode) job explicitly targets that Service so that "cluster metrics keep flowing as long as ANY node is up" (comment at helmrelease-prometheus.yaml:111-114) — but that comment's premise (jupiter is a Proxmox node that may be transiently down) no longer holds once jupiter permanently stops serving 8006. Depending on kube-proxy's endpoint selection, a fraction of scrapes against this Service will now hit a closed/non-existent port.
• kubernetes/apps/monitoring/helmrelease-prometheus.yaml:143 — the pve-node job's static target list still includes "jupiter.renner.uno" alongside neptune/saturn. This target will permanently fail to scrape (nothing serves /pve there once PVE is gone), producing a standing broken/alerting target rather than real signal.
• ansible/inventory/pve_hosts.yml:14 — jupiter remains in [pve_cluster] with ansible_user=root. This group feeds [ntp_cluster:children] (used by ansible/plays/install-ntp-ha.yml, which also hardcodes jupiter: {priority: 100, state: MASTER} at line 74-76 of that file) and is targeted directly by ansible/plays/configure-pve-k8s-zfs-tuning.yml:26 (hosts: pve_cluster, become: yes, expects SSH+ZFS on a PVE host). Both will now attempt to reach jupiter as if it were still a Proxmox host with root SSH, which is no longer true for a bare-metal Talos node.

None of these are mentioned in the PR's "Not in this PR" section (which only defers the cephcluster.yaml QLC-wear comment), so it's unclear whether they're intentionally deferred to a follow-up or simply missed. Given how carefully every other second-order effect in this PR was tracked down (the Woodpecker seed loop, the runbook, the install-disk selector), this looks like an oversight rather than a deliberate deferral, and is worth resolving or explicitly calling out before merge.

🟡 Suggestions

• kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:167-175 — the new comment candidly flags that Rook's OSD device selection (name: nvme0n1) on jupiter now depends on PCI enumeration order rather than a stable identifier, and leans on "Rook skips a disk carrying partitions" as the backstop. That backstop is real, but it's a weaker guarantee than the one just added for the Talos install disk (diskSelector.serial in k8s-1.yaml). Since the risk is already called out explicitly, consider giving the OSD device the same treatment (Rook supports selecting by /dev/disk/by-id/* path or a deviceFilter/fullpath) rather than relying on enumeration order + partition-skip as defense in depth.
• kubernetes/talos/patches/k8s-1.yaml:216-278 — I could not independently verify the LinkAliasConfig/LinkConfig/VLANConfig/DHCPv4Config/Layer2VIPConfig field names against the Talos v1.12 docs in this environment (no talosctl binary or network access here), so I can't confirm the schema beyond internal consistency (link names line up across documents, MAC selectors are per-alias, MTU/addressing matches the old per-interface config 1:1). The PR states talosctl validate --mode metal passed against v1.12.6 — worth keeping that validation output attached to the PR (or re-running it right before apply), since this is the only bare-metal control-plane node and a network-config mistake here is recovered via physical/IPMI access rather than a Proxmox console revert.

🟢 Nits

None beyond the above.

Tests

kubernetes/talos/patches/*.yaml is not part of any kustomization.yaml, so .woodpecker/ci-manifests-test.yml's kustomize-build step never touches it — this PR's highest-risk change (new Talos network schema, single control-plane node, physical hardware) has no CI coverage, same as the pre-existing k8s-2/k8s-3 patches. That's consistent with how this repo already handles Talos config (manual talosctl validate per the PR body), so not a regression introduced here, but worth flagging given the blast radius if the schema is subtly wrong. The cephcluster.yaml/storageclass.yaml changes are comment/value-only and will be caught by kustomize-build for basic YAML validity, but Rook CRD semantics (e.g. min_size behavior) aren't checked by that gate either — consistent with the rest of the file already relying on the same unchecked pattern for replicapool/data0.

Recommendation

Request changes — 5 blocking issue(s) (all variations of the same root cause: stale references to jupiter as a Proxmox node outside the files this PR touches).

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `6380f02ca4941c15953330f4aaf4d10c5fb71a97`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR converts k8s-1 (jupiter) from a Proxmox VM to bare-metal Talos, rewrites its Talos network config onto the v1.12 multi-document schema, moves the Ceph OSD to a native NVMe (U.2/PM983) drive, drops the now-orphaned `opentofu/vm_150_k8s_1.tf`, and temporarily relaxes `cephfs-metadata` to size 2/min_size 1 while the cluster runs on two remaining Ceph hosts. The reasoning in the PR body and inline comments is unusually thorough and each documented trade-off (VIP handling, NIC selection, install-disk-by-serial, the OSD-vs-system-disk enumeration risk, the metadata-pool availability trade) checks out against the code. The gap is scope: several other files in the repo still hard-wire jupiter as a Proxmox node and are not touched here, which will break once jupiter stops running PVE. ## 🔴 Blocking - **`ansible/inventory/group_vars/all/vars.yml:2`** — `pve_api_url: "https://jupiter.renner.uno:8006"` is unchanged. It's consumed by `ansible/roles/vm-update/tasks/main.yml:66` and `:82` to trigger and poll `vzdump` backups via `{{ pve_api_url }}/api2/json/nodes/{{ proxmox_node }}/vzdump` for *any* Proxmox node. Since a Proxmox cluster's API is reachable from any member and this URL is the sole entry point configured, once jupiter no longer runs PVE this breaks backup-trigger automation for the entire remaining fleet (saturn/neptune, i.e. k8s-2/k8s-3 and any other guests) — not just for jupiter's own (now-removed) VM. - **`ansible/inventory/api.proxmox.yaml:5`** — the `community.proxmox.proxmox` dynamic-inventory plugin is pointed at `https://jupiter.renner.uno:8006` as its sole `url`. Same issue: this is the only configured entry point for cluster-wide dynamic inventory discovery, and it stops answering once jupiter drops PVE. - **`kubernetes/apps/pve/endpointslice.yaml:19-23`** — jupiter is still listed as an endpoint (`10.2.40.201:8006`, `ready: true`) behind the `pve` Service. `helmrelease-prometheus.yaml`'s `pve` (cluster-mode) job explicitly targets that Service so that "cluster metrics keep flowing as long as ANY node is up" (comment at `helmrelease-prometheus.yaml:111-114`) — but that comment's premise (jupiter is a Proxmox node that may be transiently down) no longer holds once jupiter permanently stops serving 8006. Depending on kube-proxy's endpoint selection, a fraction of scrapes against this Service will now hit a closed/non-existent port. - **`kubernetes/apps/monitoring/helmrelease-prometheus.yaml:143`** — the `pve-node` job's static target list still includes `"jupiter.renner.uno"` alongside neptune/saturn. This target will permanently fail to scrape (nothing serves `/pve` there once PVE is gone), producing a standing broken/alerting target rather than real signal. - **`ansible/inventory/pve_hosts.yml:14`** — `jupiter` remains in `[pve_cluster]` with `ansible_user=root`. This group feeds `[ntp_cluster:children]` (used by `ansible/plays/install-ntp-ha.yml`, which also hardcodes `jupiter: {priority: 100, state: MASTER}` at line 74-76 of that file) and is targeted directly by `ansible/plays/configure-pve-k8s-zfs-tuning.yml:26` (`hosts: pve_cluster`, `become: yes`, expects SSH+ZFS on a PVE host). Both will now attempt to reach jupiter as if it were still a Proxmox host with root SSH, which is no longer true for a bare-metal Talos node. None of these are mentioned in the PR's "Not in this PR" section (which only defers the `cephcluster.yaml` QLC-wear comment), so it's unclear whether they're intentionally deferred to a follow-up or simply missed. Given how carefully every other second-order effect in this PR was tracked down (the Woodpecker seed loop, the runbook, the install-disk selector), this looks like an oversight rather than a deliberate deferral, and is worth resolving or explicitly calling out before merge. ## 🟡 Suggestions - **`kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:167-175`** — the new comment candidly flags that Rook's OSD device selection (`name: nvme0n1`) on jupiter now depends on PCI enumeration order rather than a stable identifier, and leans on "Rook skips a disk carrying partitions" as the backstop. That backstop is real, but it's a weaker guarantee than the one just added for the Talos install disk (`diskSelector.serial` in `k8s-1.yaml`). Since the risk is already called out explicitly, consider giving the OSD device the same treatment (Rook supports selecting by `/dev/disk/by-id/*` path or a `deviceFilter`/`fullpath`) rather than relying on enumeration order + partition-skip as defense in depth. - **`kubernetes/talos/patches/k8s-1.yaml:216-278`** — I could not independently verify the `LinkAliasConfig`/`LinkConfig`/`VLANConfig`/`DHCPv4Config`/`Layer2VIPConfig` field names against the Talos v1.12 docs in this environment (no `talosctl` binary or network access here), so I can't confirm the schema beyond internal consistency (link names line up across documents, MAC selectors are per-alias, MTU/addressing matches the old per-interface config 1:1). The PR states `talosctl validate --mode metal` passed against v1.12.6 — worth keeping that validation output attached to the PR (or re-running it right before apply), since this is the only bare-metal control-plane node and a network-config mistake here is recovered via physical/IPMI access rather than a Proxmox console revert. ## 🟢 Nits None beyond the above. ## Tests `kubernetes/talos/patches/*.yaml` is not part of any `kustomization.yaml`, so `.woodpecker/ci-manifests-test.yml`'s `kustomize-build` step never touches it — this PR's highest-risk change (new Talos network schema, single control-plane node, physical hardware) has no CI coverage, same as the pre-existing k8s-2/k8s-3 patches. That's consistent with how this repo already handles Talos config (manual `talosctl validate` per the PR body), so not a regression introduced here, but worth flagging given the blast radius if the schema is subtly wrong. The `cephcluster.yaml`/`storageclass.yaml` changes are comment/value-only and will be caught by `kustomize-build` for basic YAML validity, but Rook CRD semantics (e.g. `min_size` behavior) aren't checked by that gate either — consistent with the rest of the file already relying on the same unchecked pattern for `replicapool`/`data0`. ## Recommendation Request changes — 5 blocking issue(s) (all variations of the same root cause: stale references to jupiter as a Proxmox node outside the files this PR touches). <!-- pr-review:sha=6380f02ca4941c15953330f4aaf4d10c5fb71a97 -->

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#566