The SAN is a dedicated interface with exactly one consumer per node, so
the attachment becomes host-device: the OSD pod gets the NIC whole, with
its real MAC and MTU, instead of a macvlan child on top of it. The
device field names the role (ceph-san, the boot-time rename), so the NAD
still contains no hardware. The earlier "failed to move link" was
measured against the bridge, which cannot move — not against the NIC;
the move against the renamed NIC is untested until the next throwaway
run.
The Hetzner provisioner derives its network ranges instead of inventing
them: the cluster segment from the subnet the site patch pins node
addresses to, the storage segment from the SAN pool the attachment
allocates from. Invented ranges are how this week a fabric ended up
knowing 10.91.x while the workloads spoke 10.99.x — the same
two-statements-drift as the 10GiB/50GB disk, which already has this
rule.
Known, deliberately unresolved: the fabric reserves the first address of
a network range as its gateway, which for the storage net is the SAN
pool's first address. It matters only if cross-node SAN traffic on that
site is ever attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A machine-config udev rule renames the port to ceph-san before the
network stack opens links, so the attachment resolves a real kernel name
and the bridge indirection goes away entirely.
Measured on a plain two-NIC qemu VM booting the metal ISO built from
this cluster's own schematic, Talos v1.13.9: the rule renamed the
secondary NIC at boot, and a LinkConfig naming ceph-san found and
brought up the renamed link. #8018's "Resource busy" is the RUNTIME
case — rules applied to a running node cannot rename an open link, which
is also why rolling this out takes a reboot per node.
Match keys are machine-local, as the design demands: burned-in MAC on
the MS-01s, PCI bus path on Hetzner where the MAC is assigned at attach
time. The qemu VMs need no rule at all — their SAN is a VLAN child and
VLANConfig names it directly.
NOT rolled out to production. The rule lands inert on a running node and
takes effect at the next reboot; until then the renamed LinkConfig
matches nothing and the running link keeps its state. Per-node reboot
with drain, Hetzner first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lost with the revert of the udev experiment; unrelated to it. The warning
tested whether the sweep function is defined, which it always is, so the
qemu phase printed it too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Job cloned from in-cluster Gitea, so the rehearsal depended on the
thing it rehearses losing. A run died on resolving gitea.renner.uno
before the first VM booted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The warning tested whether the sweep function is defined, which it always
is, so the qemu phase printed it too.
Also adds a udev rule to vm-2 only, to measure whether renaming the SAN
link gives the CNI a resolvable name and removes the need for the bridge.
vm-3 and vm-4 are identical without it, so the comparison is local to one
node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comments state what holds, not how I got there. The bridge and
provisioning comments had grown into accounts of what failed first and
why I chose what I chose — commit-message material, not config.
Roughly a third of the comment lines added today are gone; the facts a
reader needs to change the file safely are kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A link alias is the shorter answer and does not reach the CNI. Talos
writes it to IFLA_IFALIAS and deliberately not as an altname — the
maintainer position is that udevd owns altnames — so a plugin resolving
its master by kernel name fails with "link not found".
siderolabs/talos#12604 is the same setup, Multus macvlan against an
aliased link, and is still open. Measured here too: a node booted with
LinkAliasConfig showed alias=ceph-san with altNames unchanged, and the
attachment failed with that error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The billing warning fires from the exit handler whenever the sweep
function is defined, so `hcloudclean` printed it too — directly after
deleting every server it could find.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The phase ended by writing node addresses to a cache file and exiting, so
the cluster build had to be a second invocation that read it. Nothing
needed that: the talos phase already takes NODES, WORKERS, K8S_API_HOST
and SITE, and the comment on those four says they are the whole of what a
test site may change.
What was wrong with the old handover was not the handover — it was
FLUX_SOURCE, HCLOUD_FULL and a ConfigMap riding along with it. Those are
gone. The four remain, as vmtest uses them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The attachment named enp2s0f1np1, a kernel interface, in configuration
every cluster shares. That forced every other site to produce an
interface with that name: the qemu VMs built a VLAN and called it that,
and its own comment admitted "the name looks physical and is not". A site
whose hardware could not be made to match had no way in at all.
Each machine now declares a ceph-san bridge over whatever its own SAN
hardware is — the second X710 port on k8s-1/2/3, enp8s0 on the Hetzner
servers, a VLAN on the qemu VMs — and the attachment names the bridge.
The interface name appears only in the file that describes the machine
that has it, which is the shape the OSD disks already use: the machine
labels itself, the CephCluster selects by label.
Aliases were the obvious route and do not work: LinkAliasConfigController
sets IFLA_IFALIAS, which is descriptive, so nothing resolves an interface
by it. A bridge is a real link with a name we choose.
The bridge repeats mtu 9000 on the production nodes. A bridge takes the
smallest MTU of its ports, but it is created before the port is enslaved,
so leaving it unset pins it at 1500 and silently caps the segment.
NOT rolled out to production by this commit. Moving a live macvlan master
onto a bridge is a per-node, drain-first change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowSchedulingOnControlPlanes and kubelet.nodeIP.validSubnets were in
every node patch. They are properties of the cluster: the first is what
shape the cluster has, the second is which segment the cluster puts node
addresses on. Both move to the site patch, where site-qemu and
site-homelab already pin their node subnets.
Interfaces, install disk, hostname, certSANs, labels and the OSD volume
stay per node. Three machines having the same NICs does not make the NICs
a cluster property — it makes them alike.
The site patch also said a cloud server has one address and therefore
nothing to pin, and that this site has no second network and no second
disk. All three stopped being true when provisioning started creating
them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enp1s0 is the public NIC, enp7s0 and enp8s0 are the private networks in
the order bootstrap.sh attaches them. Read off all three servers with
talosctl in maintenance mode and matched against the MAC addresses
Hetzner reported, rather than assumed from attachment order.
Every interface is stated, the public one included. Talos stops running
its default DHCP operators for the whole machine as soon as one new-style
network document exists, so an omitted interface gets no address at all
and the node is reachable only from the console.
kubelet.nodeIP.validSubnets pins the node address to the cluster network.
Production keeps its node IP on the management LAN, and it takes etcd,
the API and pod east-west with it; without this the kubelet can pick the
public address and every traffic class rides the internet-facing NIC.
enp8s0 carries no address, as the second X710 port does in production.
No jumbo MTU on it: 9000 is what a switched segment carries, and a
Hetzner network is not one.
Also fixed in provisioning, both found by running it: the exit trap tore
down the machines the moment the phase succeeded, and the pre-run sweep
deleted the servers the adopt path exists to reuse. Networks get one /16
each, since a server cannot be attached to two overlapping ranges, and
an existing network is reused only if its range is the one this run
wants. Server and network lookups read JSON instead of Go templates,
which compared a struct against an ID and silently matched nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The phase booked servers and then fell through into the Talos and Flux
phases as SITE=hcloud, which put site-specific branches into everything
downstream. It now provisions and exits; the cluster is built by the
ordinary phases, which carry no knowledge of this provider. The isolation
from the production cluster is printed as the caller commands rather than
set here, because both halves are needed and only the caller outlives
this phase.
Networks: two segments, matching where the traffic classes are today
rather than where they are going — the node address carries management,
cluster-internal and external alike, and Ceph replication has its own.
Attachment order is load-bearing and stated once; the run records each
segment address and MAC so the machine config can be written from what
the platform actually did instead of an assumed interface order.
Shape comes from the machine config. hcloud_volumes_for_node reads the
RawVolumeConfig out of the node patch, so the disk a node gets is the
disk Talos was told about. The size was previously stated in both places
and they disagreed: 10GiB against a 50GB volume.
Removed with it: the cluster-settings-local ConfigMap and the
CEPH_NETWORK_PROVIDER substitution, the break-glass source in
gotk-sync.yaml, the substitute-disabled annotation on the toolbox, and
the commentary I had written into the configuration files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GITHUB_REPO is "Homelab", so `flux bootstrap` writes
github.com/konairius/Homelab.git. The previous commit wrote it lowercase,
which GitHub resolves to the same repository — so nothing would have
looked broken. Flux compares strings, not repositories: the file would
have differed from what bootstrap generates on every run, been rewritten
and committed to the mirror, and the next push to Gitea would have
force-pushed that commit away again. Exactly the loop the previous commit
set out to close.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things, both about being able to iterate without paying for it.
Reuse: capacity is the binding constraint. Every node of the last run
waited through resource_unavailable before it was placed, so a machine
already in hand is worth more than a clean slate. Servers carrying this
test label are now adopted instead of created, and rebuilt from the Talos
snapshot to get back to maintenance mode — talosctl reset cannot be used
for that, because the previous run PKI is gone with its temp directory.
The OSD volume is still discarded and recreated: a reused one carries a
bluestore signature that Rook adopts, which would quietly turn a cold
rebuild into a restart test. Volumes are cheap, servers are not.
Sync source: gotk-sync.yaml now names the break-glass source it is
actually bootstrapped with on this branch. It pointed at in-cluster
Gitea, so `flux bootstrap github` had to rewrite and commit it on every
run — and that commit lives only on the GitHub mirror. The next push to
Gitea force-pushed the mirror, destroyed it, and Flux reconciled itself
back onto a Gitea URL the throwaway cluster has no secret for. A push
during a run then breaks the rebuild, which is exactly what this test
must not do. With git already matching what bootstrap generates, the
bootstrap commit is a no-op and there is nothing for a push to lose.
No comments added to gotk-sync.yaml on purpose: bootstrap regenerates
that file wholesale, so anything extra would diverge, commit, and start
the same loop again.
This file must not reach master — it already carries branch:
vmtest-flux, and now a GitHub URL as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The empty override rendered as YAML null, and Rook rejected the whole
CephCluster: "spec.network.provider: Unsupported value: null".
Quoting the placeholder in the manifest does not survive the pipeline.
Kustomize builds before Flux substitutes, and it re-emits `${VAR}` as a
bare scalar because that string needs no quotes. Substituting an empty
value into a bare scalar yields `provider:`, which is null rather than
the empty string the CRD requires.
So the ConfigMap values carry their own quote characters and the
manifest field is deliberately unquoted — quoting both would double them
into `""multus""`. Verified end to end with `flux build kustomization`
piped through `flux envsubst`, in both the multus and the empty case.
That full path is the point: the previous commit checked envsubst on its
own, which is why this was missed — kustomize runs first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An empty override can never beat a substitution default. Flux follows
POSIX parameter expansion and cannot distinguish "set to empty" from
"unset", so ${CEPH_NETWORK_PROVIDER:=multus} handed an empty value
yields multus — and ${VAR=multus} without the colon behaves identically.
Both checked with `flux envsubst`.
That mattered because the empty string is the only provider value
meaning "pod network", so the one site that needs it had no way to ask
for it. The site override was therefore inert: Hetzner would have been
handed the multus SAN it has no layer 2 to build.
Production values move into cluster-settings and the manifest carries a
bare ${VAR}, so the override wins by there being nothing to fall back to.
The cost is that deleting those keys silently moves production Ceph off
the SAN; the warning lives with them. A missing cluster-settings is not
a risk — it is non-optional in substituteFrom, so Flux fails and retries
rather than applying half-substituted values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Indented one level too deep they are not a mis-sized volume but a fatal
config: Talos rejects the whole document with "unknown keys found during
decoding", so hcloud-1 refused every apply-config and the node took no
configuration at all. hcloud-2 and hcloud-3 were unaffected — the drift
came in with the comments explaining the selector.
Also corrects the volume size in that comment, which still said 20GB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things stood between the Hetzner rehearsal and a cluster that comes up, and
they had to be solved together rather than one run at a time.
DISKS. The nodes had none beyond their system disk, so ceph-block existed as a
StorageClass with nothing behind it and OpenBao's PVC sat Pending. Each node now
gets a 50GB volume, attached raw — not formatted, not automounted, because Rook
needs the bare device and a filesystem makes Ceph skip the disk as in use — and
carries the storage.renner.uno/ceph-osd label the CephCluster selects on. That
label is why this works at all: OSD nodes are chosen by label rather than
hostname, so a cluster built elsewhere gets OSDs.
50GB, not the 10GB minimum I first wrote. The reference is the qemu site, which
gives each OSD worker 24GiB and gets twelve app Kustomizations up; production's
2.2TB is years of real data that neither site restores. 10GB would leave ~10GB
usable across three replicas — less than OpenBao's two PVCs ask for.
The disk selector drops the `disk.transport == "virtio"` clause the qemu
patches carry. A Hetzner volume arrives over virtio-scsi, and a selector
matching zero disks is skipped silently — the OSD would simply never appear.
NETWORK. The CephCluster demands multus with the ceph-san macvlan attachment,
whose master is the kernel link enp2s0f1np1. That cannot exist on Hetzner:
their networks are layer 3 only — "the servers are not directly connected with
each other, instead they are connected via a gateway" — so there is no layer 2
for macvlan and no VLAN to manufacture the name.
Rather than fork the manifests, provider and selector become substituted values
defaulting to what production runs. The override arrives in a
cluster-settings-local ConfigMap that exists only on the throwaway cluster,
listed second and `optional: true` so production is untouched. An empty provider
is a valid CRD value and puts Ceph on the pod network.
That required enabling postBuild substitution on a path that never had it, and
toolbox.yaml is full of shell ${MON_CONFIG}-style variables that Flux would
replace with empty strings. It carries
kustomize.toolkit.fluxcd.io/substitute: disabled for exactly that reason —
without it the toolbox would write an empty ceph.conf and never find a monitor.
The break-glass bootstrap died on
GET https://api.github.com/repos/konairius/Homelab: 401 Bad credentials
after the cluster was already built. The token entry read
${GITHUB_TOKEN:-${GITEA_TOKEN:-}} — whichever variable happened to be set. On
this workstation GITEA_TOKEN is exported and GITHUB_TOKEN is not, so a run with
FLUX_SOURCE=github handed a Gitea token to the GitHub API.
The comment directly above it already said "which token depends on FLUX_SOURCE,
and that is not a detail". The code did not read it. git_token_for_source now
chooses by the forge that will actually be authenticated against, with
`gh auth token` as the GitHub fallback — the credential the operator already
has here, and the one the vmtest secret feeder has always used.
Also removes the two `2>/dev/null` I had written today.
- gh is called plainly. Swallowing its stderr would turn a failed login into
an empty string, and require_secrets would report "came back empty" for a
credential that exists but could not be read. Its non-zero exit propagates
instead, which is what makes the reason visible.
- the firewall address lookup keeps both curl attempts' stderr and prints it
if BOTH families fail. One failing is ordinary; both failing is a fault
whose cause is the diagnosis.
Nine `2>/dev/null` remain in this file, all older than today. Left alone here
rather than swept up mid-test, but they are the same hazard: each one makes a
zero indistinguishable from a failure.
Two faults, one found by the provider and one by me.
RETRY ON PLACEMENT. The first cx43 run died on
hcloud: error during placement (resource_unavailable, ...)
which is Hetzner being momentarily out of that type in that location, not a
wrong request — probing the same type in the same location minutes later
created a server fine. For a test meant to run on a schedule that distinction
is the whole difference between a real failure and a red run nobody needs to
look at. Placement errors are now retried for ten minutes; anything else still
fails immediately, with the provider's message rather than a bare exit code.
KEEP MUST NOT SUPPRESS THE SWEEP. cleanup_hcloudtest has two callers: the sweep
BEFORE a run, which exists so a previous crash cannot leak servers, and the
teardown after it. I put the HCLOUD_KEEP check inside the function, which
disabled both — so a kept cluster would survive into the next run, and that run
would then fail creating servers whose names were already taken. The check
moves to on_exit, the only caller that means "after".
Also drops the talosconfig hint from the keep message: it printed "?" because
the variable is not in scope there, and a placeholder is worse than saying
nothing.
cx23 has 4 GB, 3.2 GB of it allocatable. The three nodes sat at 59-78% of that
with the infrastructure skeleton alone — no Ceph OSDs, no OpenBao running, not
one app — with limits already at 134-207% of allocatable and both
kube-controller-manager and kube-scheduler restarted twice. There was no room
for the thing the cluster is built to run.
I chose it as the successor to the cx22 the Talos Hetzner guide names, which is
a minimal example rather than a sizing statement for this workload. That is the
same mistake I had already made and corrected today on the local control plane,
which went 6GiB → 8GiB → 16GiB before it stopped evicting its own API server.
cx43 instead: 16 GB, 8 vCPU. The local rehearsal is the reference — a 16GiB
control plane and three 8GiB workers, 40GiB in total — and three nodes here
carry that same workload plus a control plane each, so 48GiB is the comparable
figure.
Constraints that narrowed it: cpx11/cpx21 are US-only, and cax* are ARM, which
would not match the amd64 schematic this cluster installs.
require_secrets picks its branch on whether OPENBAO_SECRETS_DIR is set — fetch
from pass when it is empty, wait for streamed-in files when it is not. The
function sets that variable itself, so the test was reading state it had
already changed.
The subset support added for the talos phase made that visible. hcloudtest asks
for ghcr-pull and hcloud-token before anything else; from then on the variable
was populated, and the full call for the flux phase took the wait branch. On a
workstation with pass right there, the run printed streaming instructions
naming a `kubectl -n vmtest exec` Job that does not exist, and stopped — while
three Hetzner nodes sat in maintenance because the wait comes before the talos
phase applies any config.
SECRETS_SELF_FETCH records which world the run is in, decided once on first
entry: an empty directory means pass is available, an inherited one means the
in-cluster Job. Later calls follow that decision rather than re-deriving it.
The fetch loop also skips what a previous call already collected. A subset call
fetches only what it asked for, so the full call after it fills the gaps — and
re-running `pass show` for a key already on disk is how a locked store prompts
twice and an unattended run hangs.
The three-node run provisioned a cluster, watched it form, and destroyed it
without anything having run on it. Provisioning is the first step of the task,
not the task.
HCLOUD_FULL=1 continues into flux and wait-apps instead of stopping after
Talos, so the run reaches whatever actually blocks it rather than the blocker
being guessed at in advance. It needs all six secrets, including the OpenBao
unseal key — which works from a workstation, where require_secrets reads pass,
and is exactly what does not work unattended. That is why it is a switch and
not the default.
HCLOUD_KEEP=1 leaves the servers up. The teardown fires the moment a run ends,
including when it ends badly, which is precisely when the machines are worth
looking at — the same reasoning as VMTEST_KEEP, with the difference that these
bill by the hour and nothing else reclaims them. The message says so and names
`bootstrap.sh hcloudclean`.
FLUX_SOURCE is forced to github in this phase, not defaulted: gitea.renner.uno
has no external ingress, so from a Hetzner node it does not answer at all. The
break-glass source is the only one this site has — which is the position a real
recovery is in, and the reason the test is worth running from outside.
The three-node Hetzner run printed
==> Waiting for K8s nodes...
Talos phase complete.
with nothing between, and a kubectl I ran seconds earlier had listed no nodes at
all. Whether the wait saw three Ready nodes in that gap or matched zero against
zero cannot be decided from the log, because the step is silent on success.
That is the defect: a check nobody can audit is not much of a check.
It now prints the count it accepted, which is the same number its pass
condition uses.
And it refuses to run with an empty expectation. `expected` comes from
ALL_NODES, handed over by whichever phase provisioned the machines; if that
array were ever empty, `0 -eq 0` satisfies the loop's exit condition on the
first pass and the script reports a complete Talos phase for a cluster with no
nodes in it. An empty ALL_NODES is a bug in the provisioning phase, not a
cluster that is ready, so it fails loudly instead.
Neither change alters what a correct run does. Both change what a broken one
looks like.
One machine is a node. etcd has no quorum to lose, nothing schedules anywhere
else, and a rebuild that only ever ran on a single box proves nothing about the
one that has three. The phase now provisions three control planes, all carrying
workloads — which is the production shape: k8s-1/2/3 are all control planes and
all run apps.
The qemu site cannot mirror that. Its provisioner hands a second disk to
workers only, so a Ceph OSD forces 1 control plane and 3 workers there. Hetzner
attaches volumes to any server, so this site can match production instead of
the emulator. allowSchedulingOnControlPlanes moves to the site patch, where a
cluster-wide property belongs, and the node patches carry only what is true of
each machine.
Also opens the firewall between the nodes, which one node did not need and
three do: etcd peers on 2380, the apiserver, kubelet on 10250 and Cilium's
VXLAN all run between them, and with only the this-host rules etcd never forms
a quorum. The rule is added after creation because the addresses do not exist
before it, and it admits all ports between the members rather than an
enumerated list — guessing the full port set of Talos plus Cilium is how a
rebuild fails on the one port nobody wrote down. The outside still sees 50000
and 6443, and only from the host running this.
Not yet run at three nodes. The single-node run before this reached "Talos
phase complete" with the API answering and Cilium installed, and left nothing
behind.
26092ba5 changed the default to "hetzner" and justified it with "no such
profile exists on the workstation". That is false. Another session ran this
script with the original hardcoded "secrets" and it succeeded, and both
"secrets" and "hetzner" are present in ~/.aws/credentials here.
So the default goes back to "secrets" — unchanged behaviour for everyone it
already worked for, and the owner's long-standing choice. Changing it on the
strength of one failed run in one environment risked breaking the sessions
where it works, which is exactly what the previous commit did.
What was genuinely broken stays fixed: the assignment was unconditional, so it
overrode a caller who had already selected a profile, and defeated the
env-credential path this file's own header documents. `${AWS_PROFILE:-secrets}`
keeps the default and restores the override.
My own 403 therefore had some other cause in this session's environment, which
I have not identified and am not going to guess at in a commit message.
import-cluster-access.sh hardcoded AWS_PROFILE="secrets". No such profile
exists on the workstation; the one holding the Hetzner Object Storage keypair
is "hetzner". The failure this produced is worse than a missing profile
usually is:
fatal error: An error occurred (403) when calling the HeadObject
operation: Forbidden
which reads as a permissions problem on the bucket, and sends you looking at
bucket policy rather than at a profile that was never there.
`${AWS_PROFILE:-hetzner}` rather than a second hardcoded name, so a caller can
override it — the header has always documented env credentials as an equally
valid path, and the unconditional export silently defeated that.
Found while restoring ~/.kube/config and ~/.talos/config after the hcloudtest
phase overwrote them (fixed in 174e0862). Verified end to end: both files
restored, and the real cluster answers with all three control planes.
Two faults the first real runs exposed, one of them damaging.
ISOLATION. The phase exported TALOSCONFIG and KUBECONFIG to a temp directory
and stopped there. But TALOSCONFIG_PATH and KUBECONFIG_PATH are derived at the
top of this script, from those variables, before any phase runs — so exporting
them afterwards moves nothing, and the talos phase writes to the _PATH ones.
The throwaway cluster's credentials were written over ~/.kube/config and
~/.talos/config, and another session working on the real cluster lost its view
of it mid-task. The vmtest phase has always set both halves; this one now does
too.
BOOTSTRAP RETRY. `talosctl bootstrap` was called once, and anything that was
not "already exists" was reported and then ignored. "bootstrap is not available
yet" is a moment, not a fault: the preceding wait only establishes that the node
answers with its hostname, while etcd accepting the call comes later and how
much later depends on the machine. On qemu the gap is short enough to miss. On
a Hetzner cloud server it is not — the single attempt failed with
FailedPrecondition, the run carried on, and then sat out its full timeout
waiting for an apiserver that could never start because etcd had never been
bootstrapped. It now retries for ten minutes, treats "already exists" as
success, and fails loudly if the call never lands.
Also adds a check that the generated registry patch is a readable file, after
a bug where it was not: require_secrets prints "All secrets present." on
stdout, and calling it inside registry_auth_patch made that line part of the
captured path. talosctl reported `open All secrets present. | /tmp/…: no such
file` sixty times while the node sat in maintenance. The call moved out of the
substitution; a function whose value is its stdout must be the only thing
writing there.
Building the snapshot exercised the API for real and found the server type
wrong. The Talos Hetzner guide names cx22 and `hcloud server create` rejects it
outright — the type is gone. In the German locations the current shared x86
types are cpx12 and cx23; cpx11/cpx21 are US-only, and cax* are ARM, which
would not match the amd64 schematic this cluster installs. Default is now cx23.
The snapshot the phase requires exists: image 427962556, labelled os=talos,
built from factory.talos.dev with this cluster's own schematic and version read
out of cluster.yaml rather than pinned separately —
factory.talos.dev/image/0f6fb8bd.../v1.13.9/hcloud-amd64.raw.xz
written to /dev/sda in the rescue system, which produced the expected four-
partition Talos layout. Verified findable by exactly the query the phase runs.
Everything used to build it was removed afterwards: the builder server, the
ephemeral SSH key on both ends. The only server left in the project is
netbird.renner.uno, which was never touched and is delete-protected.
The token is now in pass, so the phase could be checked against the real API
for the first time — and the first thing it showed is that this project is not
empty. netbird.renner.uno has been running in it for 196 days and is the VPN
way in.
The label was the only thing separating a throwaway server from it, and
`hcloud server list -l ""` lists EVERY server, so an unset HCLOUD_LABEL would
have turned the sweep into a project wipe. Verified read-only that the selector
matches nothing today, which is correct but is a property of one variable
holding its value.
Two independent conditions now: the label is checked for emptiness before it is
used, and every candidate must additionally carry the name prefix this phase
assigns. A server that has the label but not the prefix is reported and left
alone rather than deleted.
Also corrects the token path. It is homelab/hetzner/cloud-api-secret — spelled
correctly, next to the pre-existing homelab/hetztner/ that holds the restore
keys. Two folders, one typo, both real; the comment says so, because the
obvious "fix" of unifying them would break whichever path was not moved.
The structural version of this is a separate Hetzner project with a token
scoped to it, so a bug here cannot reach production at all. Noted in the code;
worth doing before this runs unattended.
Still not executed: no snapshot labelled os=talos exists yet, which is the last
prerequisite and the one the phase fails on with the recipe.
The vmtest phase cannot test the one thing the registry mirror exists for. The
nodes carry machine.registries.mirrors with gitea.renner.uno first and ghcr.io
second, and the second endpoint only matters in the window where the first does
not exist. At home the first always exists — the production cluster is running
and answers on the LAN — so containerd never falls through and the fallback
stays unproven. From Hetzner, Gitea does not answer at all: it has no external
ingress.
The phase mirrors vmtest exactly: it provisions, sets the five variables the
talos phase reads, and falls through into it. A rehearsal that runs different
code proves less than one that does not.
Teardown is the part that had to be right, because this one bills by the hour.
It runs from the single EXIT trap, and it deletes by LABEL rather than by
remembered ID — so a run killed between `server create` and its own bookkeeping
is still cleanable, by the next run (which sweeps before it creates) or by
`bootstrap.sh hcloudclean`, which does nothing else.
The Talos API is unauthenticated in maintenance mode, which is how apply-config
reaches a node with no config. On a public address that is closed to everything
but the runner: the phase learns its own public IP and creates a firewall
admitting 50000 and 6443 from that address alone.
Scope, and why it is not the full topology: Hetzner is not the limit — a cloud
server takes up to 16 volumes and 3 private networks, so extra disks and NICs
are available. ceph-san is, because it binds to the kernel link name
enp2s0f1np1 and macvlan resolves its master by kernel name; LinkAliasConfig was
already tried on a throwaway node and does not help (vm-2.yaml). site-qemu
manufactures the name with a VLAN interface, and whether a Hetzner private
network passes VLAN tags is unverified, so it is not assumed. Everything
downstream of Multus therefore stays out, as does Flux — that phase needs the
OpenBao unseal key, which deliberately never reaches CI, and this test is meant
to run unattended.
The Talos snapshot is a prerequisite rather than something this builds; the
error names the exact one-off recipe, with this cluster's schematic and version
substituted from cluster.yaml rather than left as placeholders.
NOT YET RUN. hcloud is not installed here, there is no Hetzner Cloud token in
pass, and no snapshot exists, so none of this has been executed — only parsed.
The three prerequisites are what the first run will find.
require_secrets is gated on `phase_runs flux`. The node registry credential I
added in 2946dfcd is read in the TALOS phase, so `bootstrap.sh talos` on its own
— documented as "the part that can run unattended", and the default the vmtest
job uses without VMTEST_FULL — would reach registry_auth_patch with no secrets
collected and fail on a file nobody had asked for.
It went unnoticed because every run since has set VMTEST_FULL=1, which pulls in
the flux phase and therefore the full require_secrets first.
require_secrets now takes an optional list of names, and the talos phase asks
for `ghcr-pull` alone. Demanding the whole list there would be worse than the
bug: it would make an unattended talos run depend on the OpenBao unseal key,
which is deliberately kept out of CI — the reason the secret streaming is
manual in the first place.
The subset is validated against the list, so a typo in a name fails loudly
instead of silently demanding nothing.
creationPolicy: Merge cannot create a Secret. It only writes into one that
already exists, on the assumption that `flux bootstrap` made it — and that
assumption fails in precisely the situation this cluster gets rebuilt from.
The break-glass bootstrap pulls from the GitHub mirror and keeps its token in
flux-system-breakglass, so `flux-system` is never created. This ExternalSecret
then has nothing to merge into, produces no Secret, and still reports
Ready=True. Observed on a rebuild rehearsal: nine of nine ExternalSecrets Ready,
`flux-system` absent, and the moment anything pointed the GitRepository at it:
GitOperationFailed: failed to get secret 'flux-system/flux-system':
secrets "flux-system" not found
So the documented handover back to Gitea — switch the source once Gitea is
serving — could not be performed at all. The credential it needs was never
going to be there.
Orphan creates the Secret when absent and updates it when present, without an
ownerReference. That is what the header comment already claimed to want:
losing this object, or a failed sync, leaves Flux's last-good credential in
place. Owner would have created it too, but with an ownerReference that takes
the credential down with the ExternalSecret.
Safe on the live cluster: `flux-system` holds exactly the two keys written
here — verified, username and password — so writing it whole and merging into
it produce the same object. deletionPolicy is stated explicitly because it is
the half that matters: if the OpenBao entry disappears, keep serving what Flux
already has.
Upstream will not solve this for us. fluxcd/flux2#2761 asked for a fallback URL
on GitRepository and was rejected, with the reason that lands exactly here —
"the deploy keys are unique per repo so the auth will fail, same with HTTPS
access" (stefanprodan). A single source with two credentials is the shape Flux
has, so the credentials have to be kept apart by us.
Four images are needed while the cluster is still being built: the CNI plugins
every pod depends on, paperless's fetch-bridge-cert init container,
zigbee2mqtt's seed-data init container and the trmnl renderer. None of them can
come from gitea.renner.uno during a rebuild, because Gitea runs inside the
cluster being rebuilt.
cni-plugins solved that by referencing ghcr.io directly, and the earlier commits
in this branch were about to do the same to the other three. That fixes the
rebuild and breaks the normal case: every steady-state pull then leaves the
network for an image the local registry already holds. The recovery source and
the everyday source are different things — the same distinction Flux's git
source makes between the in-cluster Gitea and the GitHub break-glass mirror.
So the fallback moves into machine.registries.mirrors, where containerd takes an
ordered endpoint list: gitea.renner.uno first, ghcr.io second. Normal operation
is unchanged and local; ghcr answers only in the window where the first cannot.
No manifest names an external registry any more, cni-plugins included.
Containerd keeps the repository path when it falls through, so both registries
have to agree on it. `gitea_admin/alpine-jq` cannot be served by ghcr, where the
namespace is fixed to the account name — hence the konairius/ path on both, one
digest pushed to each by the same build. Verified equal on this run:
Pushed gitea.renner.uno/konairius/alpine-jq@sha256:87d0bf88...
Pushed ghcr.io/konairius/alpine-jq@sha256:87d0bf88...
The ghcr namespace stays private — three of the images this will eventually
cover are own applications, not alpine+jq — so the node also carries a
credential, in machine.registries.config, read from pass as the sixth required
secret. An imagePullSecret cannot work here: it would need to exist in five
namespaces before any of them does, and it would come from OpenBao, which needs
Ceph, which needs these very images.
Production still needs the machine config applied before a rebuild relies on
it; in steady state the references resolve from Gitea with or without it.
The previous commit put alpine-jq, alpine-openssl and cni-plugins on ghcr.io so
a rebuilding cluster can reach them. Pointing the manifests there was the wrong
conclusion: normal operation must pull from the local registry, and an external
reference sends every steady-state pull out of the network. The recovery source
and the everyday source are different things — the same distinction Flux's git
source already makes between the in-cluster Gitea and the GitHub break-glass
mirror.
Talos expresses that with machine.registries.mirrors and an ordered endpoint
list: gitea first, ghcr second. Containerd keeps the repository path when it
falls through to the next endpoint, so both registries have to agree on it.
`gitea_admin/alpine-jq` cannot be served by ghcr, where the namespace is fixed
to the account name.
Hence a second Gitea destination under konairius/, matching the ghcr path
exactly. The manifests will reference that one; the mirror makes it resolve
during a rebuild without any of them naming an external registry.
Pipeline only, again. The images do not exist under the new path until this
runs, so the references move in a separate commit afterwards. The gitea_admin/
copies stay until nothing points at them.
Needs the konairius org in Gitea (created) and package write for the pipeline's
token in it — this run will show whether ci_images_token already has it.
Three images are needed while the cluster is still being built — paperless's
fetch-bridge-cert init container, zigbee2mqtt's seed-data init container and
the trmnl renderer. None of them can come from gitea.renner.uno, which runs
inside the cluster being rebuilt; a rehearsal left all three in ImagePullBackOff
and took their Kustomizations down with them.
They are mirrored to ghcr.io (previous commit) but the namespace is private,
and it stays private: three of the five images this will eventually cover are
own application images, not alpine+jq. Making them world-readable is not
something the rebuild requires.
The credential goes into machine.registries.config, not an imagePullSecret.
An imagePullSecret fails here twice over: it would have to exist in five
namespaces before any of them exists, and it would come from OpenBao, which
needs Ceph, which needs these very images. containerd's own auth is in place
before the first pod is scheduled and applies to every pull on the node
regardless of namespace (pkg/machinery/config/types/v1alpha1,
RegistryAuthConfig — field names follow .docker/config.json).
The token is the sixth entry in OPENBAO_REQUIRED_SECRETS, so a run demands it
up front rather than discovering it missing after the disks are wiped. The
generated patch lives in the in-memory secrets dir and dies with the Job.
Only alpine-jq and alpine-openssl move in this commit, because only they are
published to the mirror so far. vsftpd, shelly-manager and
authentik-samba-bridge-ldap-outpost are built by other repositories and need
the same push there first; their references stay on gitea until then.
Production rollout is NOT this commit. The nodes need machine.registries
applied before any manifest points at ghcr, or the pods cannot pull. Order:
apply the machine config, verify a pull, then move the references.
cni-plugins has been publishing to ghcr.io since a cold-rebuild rehearsal
showed that gitea.renner.uno cannot serve a cluster that does not exist yet.
The comment there claimed it was "the only image here that must". A later
rehearsal disproved that: paperless sat in Init:ErrImagePull on
alpine-openssl, and trmnl's renderer on alpine-jq, both for the identical
reason.
The rule is not about any one image. Anything an app needs while the cluster
is still being built cannot live only in that cluster's own registry —
zigbee2mqtt's seed-data init container, the trmnl renderer and paperless's
fetch-bridge-cert init container all start during the rebuild.
Gitea stays the primary destination; ghcr.io is the copy a bare cluster can
reach. It has to stay credential-free to pull, because an imagePullSecret
would come from OpenBao, which needs Ceph, which needs the cluster to be up.
Pipeline only. The manifests still reference gitea.renner.uno and will keep
working; they move to ghcr in a separate commit once the images are actually
published there, because switching the references first would break the apps
immediately.
Needs the ghcr_mirror_token secret, now present as a global Woodpecker secret
scoped to write:packages and available on push and deployment — the two events
this pipeline runs on.
kubernetes/apps/home-assistant/ha-pg-objectstore.yaml declares an ObjectStore,
whose CRD comes from the CNPG Barman plugin in infrastructure-cnpg-barman.
The Kustomization never said so, and on a cold cluster that means the kind is
unknown when the path is first applied. Flux's server-side dry-run then
rejects the whole set over the single object it cannot recognise:
ObjectStore/home-assistant/ha-pg-store dry-run failed:
no matches for kind "ObjectStore" in version ...
Six app paths declare an ObjectStore — authentik, home-assistant, immich,
paperless, trmnl, woodpecker. Five of them already list the dependency. This
was the one that did not.
The blast radius is four Kustomizations, not one: speech-to-phrase and
zone-configurator depend on home-assistant directly, and matter-server health-
checks a Deployment inside it, so all three stalled behind an ObjectStore none
of them uses.
The csi.ceph.io CRDs come from the ceph-csi-operator subchart inside the
rook-ceph HelmRelease, which runs with `crds: CreateReplace` and `retries: 3`.
They are therefore not created once — they are created again on every attempt,
and every custom resource living under them goes with them.
A cold rebuild landed squarely in that window:
19:24:21 ceph-csi-drivers install succeeded — both Driver CRs created
19:25:50 csi.ceph.io CRDs recreated by a later operator attempt
19:29:33 rook-ceph-operator install finally succeeded
Helm still had the two Driver objects in its stored release, so nothing
re-rendered them and nothing reported them missing. The ceph-csi-operator sat
idle with no Drivers to reconcile, no provisioner Deployment was created, and
every PVC on ceph-block stayed Pending on "waiting for a volume to be created
by the external provisioner rook-ceph.rbd.csi.ceph.com". openbao-0 never got a
node, and the run failed 15 minutes later reporting that OpenBao would not
answer — five layers from the cause, with Ceph itself reporting Ready and
HEALTH_OK throughout.
This file previously argued against ordering it: "nothing here installs those
CRDs, so a dependsOn would only move the wait". That holds for the CRDs
existing, which was the failure being fixed at the time. It does not hold for
them being replaced, which is a different event and the one that bites here.
dependsOn on rook-ceph-operator waits for the last attempt instead of the
first.
The install retries stay as a backstop: a CRD replace outside an install of
this release would still empty it, and a retry is cheaper than waiting out the
30-minute interval.
6GiB was not enough and neither was 8: at both sizes vm-1 reached 99% of
allocatable memory once the infrastructure layer was up. A control plane with
nothing left over stops being one — the kubelet admits the node-critical CNI
DaemonSets by evicting whatever ranks below them, and kube-apiserver, a static
pod at system-cluster-critical, is what ranks below them:
message: Pod was preempted by Kubelet to accommodate a critical pod.
reason: Preempting
That was misread twice as a priority problem. It is not. The ordering is
deliberate and correct — a node without its CNI cannot run anything — and
nothing about it blocks: the memory shortfall blocks, and priority only picks
who pays for it. Run 26 showed this cleanly inside a single run: with the CNI
demoted to cluster-critical the API server survived bootstrap, and it was
evicted within a minute of the DaemonSets going back to node-critical, on an
unchanged node. Same pressure, different victim.
So the control plane gets 16GiB and the Job's ceiling moves with it (40G of
guests). The hosts carry ~95GiB each with 43-66GiB free, so this costs
nothing that matters.
vm-1 is the only control plane and still takes workload, so it carries the
static pods, the CNI and its share of the infrastructure layer at once. That
is the asymmetry with production, where every node is a control plane and none
of them is small.
This reverts the priority classes set in ce31dace and the wording in
d0de3798, putting multus, whereabouts and cni-plugins-installer back on
system-node-critical.
Demoting them was the wrong move. The class name says what a pod is, and
these are node-critical by what they do: without multus a node cannot attach
a secondary network, and only that node is affected — nothing cluster-wide
fails. Cilium and cni-dhcp-daemon are in the same class for the same reason.
Choosing cluster-critical instead picked a number for its ordering and made
the manifest state something untrue about the component.
It also aimed at the wrong thing. The kubelet evicted kube-apiserver because
the control plane had run out of memory; priority only decided which pod was
chosen once that had happened. Equal priority took the API server out of the
candidate set without removing a single byte of the pressure — the next
victim is simply someone else, or the CNI pod stalls at Pending, which is the
defect 0b76c130 set out to fix.
So the preemption stays open, and it is open where it belongs: the control
plane is starved during a rebuild, when the whole infrastructure layer lands
on it at once. That is fixed with memory or with kubeReserved/systemReserved
in the machine config, not by reordering the components on top of it.
Recorded next to the sizing flags in bootstrap.sh.
The comment added in ce31dace claimed "preemption is not a capacity
question". That inverts the cause. The kubelet preempts because the node is
out of memory; without a shortfall nothing is evicted, whatever the priorities
say. Priority only selects the victim once the shortfall exists.
What the two runs isolate is exactly that and no more: at 8GiB with
node-critical the API server was chosen, at 8GiB with cluster-critical it was
not. The pressure did not change — the candidate set did. Under pressure the
kubelet now displaces an ordinary workload instead, or this pod stays Pending,
which is the original defect and the acceptable one of the two outcomes.
Also records why equal priority is enough: Preemptable() compares with a
strict >, and the API server is a static pod, so IsCriticalPod holds for it
and the critical-beats-noncritical branch never applies.
0b76c130 gave the three secondary-network DaemonSets system-node-critical,
copying cilium and cni-dhcp-daemon. That is 2000001000, and Talos runs
kube-apiserver as a static pod at system-cluster-critical, 2000000000 — so
the change made the CNI outrank the control plane. On a full node the kubelet
did exactly what that ordering tells it to:
message: Pod was preempted by Kubelet to accommodate a critical pod.
reason: Preempting
phase: Failed
Both runs after the change died that way, and 097a0fac's larger control plane
did not help: it reproduced unchanged at 8GiB. During a rebuild the whole
infrastructure layer lands on a single-node control plane at once, and in that
window the node is full whatever its size. Preemption is an ordering, not a
capacity question, and no amount of memory makes a lower-priority pod safe
from a higher-priority one.
system-cluster-critical instead. Equal priority cannot preempt, so the API
server is out of reach of these pods. What 0b76c130 was for still holds:
2000000000 is far above any ordinary workload, so a rollout onto a busy node
displaces application pods rather than stalling at Pending forever — which was
the original defect, where multus sat unscheduled and every secondary network
on that node silently stopped working.
The residual is symmetric: the API server cannot displace multus either. That
is the correct trade when the alternative is losing the control plane.
The 8GiB control plane in bootstrap.sh stays, on its own merits — at 6GiB it
sat at 99% of allocatable memory with nothing left for late arrivals — but its
comment no longer claims to fix the preemption, because it never did.
Note: cni-dhcp-daemon.yaml still carries system-node-critical, from long
before this. It is the same hazard and is left alone here rather than widened
into unrequested scope.
Making the CNI DaemonSets system-node-critical (0b76c130) had a consequence I
did not account for. That class is 2000001000; Talos runs kube-apiserver as a
static pod at system-cluster-critical, 2000000000. The CNI pods therefore
outrank the API server, and on a node with no headroom the kubelet resolves
the next admission by evicting it.
That is what happened on the first run after the change. The 6GiB control
plane was at 99% of allocatable memory once the infrastructure layer was up;
the kubelet logged
preemption/preemption.go:127 "Pod evicted successfully"
pod: kube-apiserver-vm-1
and the API server never came back, because the room it needed never freed.
The run reported "openbao-0 never answered bao status" three steps later,
which is true and useless — the pod could not answer because there was no API
server to reach it through.
The priority ordering stays. A node without its CNI cannot run anything,
which is why cilium and cni-dhcp-daemon already carry the same class. What it
demands in return is capacity: a control plane must keep enough room that
admitting a node-critical pod never has to displace the control plane itself.
6GiB did not, so the throwaway control plane gets 8GiB, and the Job's own
ceiling moves with it (32G of guests now, not 30).
The obligation is written down in both places it can be violated from —
next to the sizing flags in bootstrap.sh, and next to the priority class in
multus-daemonset.yaml.
The GitHub break-glass bootstrap let `flux bootstrap` use its default secret
name, `flux-system`. That is also the target of the flux-system-git-auth
ExternalSecret in post-crds, which carries the GITEA credentials — correct for
the steady state, where the source is the in-cluster Gitea.
Sharing the name means that the moment infrastructure-crds reconciles and ESO
resolves, the GitHub token is replaced with Gitea credentials while the
GitRepository still points at github.com. Flux immediately loses the ability
to read its own source:
GitOperationFailed: unable to list remote for
'https://github.com/konairius/Homelab.git': authentication required:
Invalid username or token
So the cold rebuild destroys its git source at the exact moment it starts
working. The better the run goes, the sooner it kills itself — which is why
this stayed hidden through twenty-two rebuild attempts: not one of them had
ever got the ExternalSecrets to resolve.
--secret-name=flux-system-breakglass keeps the two apart. `flux-system` still
receives the Gitea credentials from OpenBao and simply goes unused until the
source is pointed back at Gitea, which is already a deliberate manual step
(RUNBOOK-cold-rebuild.md), announced by the warning printed right above this
call.
Nothing in the steady-state path changes: the Gitea bootstrap keeps the
default name, and the committed gotk-sync.yaml keeps referencing
`flux-system`.
Three of the four DaemonSets that make up the secondary-network path ran at
priority 0. cni-dhcp-daemon.yaml, sitting in the same directory, has been
system-node-critical all along, and so is cilium's own DaemonSet — multus,
whereabouts and cni-plugins-installer were the odd ones out.
At priority 0 they cannot preempt anything. Any event that forces a fresh
scheduling decision — a rollout, a node restart, an eviction — has to fit the
pod onto a node that may since have filled up, and if it does not fit the pod
just stays Pending. The node then has no multus, or no IPAM, or no macvlan
binary, and nothing reports it: pods still start, they simply never receive
their secondary attachment.
Found on a cold rebuild. Two of four nodes sat at 99% memory requests when the
DaemonSet rolled; both new multus pods went Pending with "1 Insufficient
memory, 3 node(s) didn't satisfy plugin(s) [NodeAffinity]" and stayed there.
Rook's network canary on one of those nodes hung in Init:0/1 in
wait-for-network-status-annotation — waiting for an annotation multus was not
running to write. Its CephCluster reconcile stopped at "discovering ceph
cluster network CIDR(s)" and never returned to status.phase Ready, which is
exactly what infrastructure-ceph-cluster's healthCheckExpr gates on, so the
entire apps layer stalled behind a scheduling decision three layers away.
cni-plugins-installer.yaml already carries a comment describing this same
canary-hang, reached from a different cause (the plugin image was pulled from
the in-cluster registry, which does not exist during a rebuild). Same symptom,
same four-layer distance between cause and evidence.
A cold rebuild deadlocked with every app Kustomization behind it:
infrastructure-ceph-cluster → CephCluster status.phase == Ready
→ Rook's node-daemon reconcile settles
→ monitoring.coreos.com CRDs exist
→ kube-prometheus-stack (monitoring)
→ grafana-db
→ infrastructure-ceph-cluster
Nothing outside that loop can break it. The operator logged
"node reconcile failed: failed to enable service monitor: the server could
not find the requested resource (post servicemonitors.monitoring.coreos.com)"
every two minutes while the CephCluster sat in Progressing, and
infrastructure-ceph-cluster's healthCheckExpr waits on exactly that phase.
Rook treats the two ServiceMonitors differently, and only one of them is
fatal (v1.20.7):
- mgr.go:344 logs and continues — "we don't want to return an error to
block the cluster reconcile since monitoring is an optional service"
- nodedaemon/reconcile.go:254 returns the error, which aborts the whole
per-node reconcile and requeues it forever
So the exporter's ServiceMonitor, the one Rook considers mandatory, is what
held the cluster.
`metricsDisabled` is not the switch to reach for — that turns the metrics
themselves off. `monitoring.enabled: false` only stops Rook from declaring
the scrape targets: configurePrometheusModule is gated on !MetricsDisabled
alone, so the mgr prometheus module and the ceph-exporter DaemonSet keep
running and serving exactly as before.
Both objects are reproduced in apps/monitoring-rules field for field off the
live production cluster — same names, namespace, selectors, ports, 10s
interval and `cluster` relabeling. The only thing lost is the CephCluster
ownerReference, which Flux cannot set.
This is the third component to need this treatment, after velero and cilium,
and the same sentence covers all three: a chart or operator that renders a
monitoring.coreos.com object at install time cannot sit below the layer that
provides those CRDs.
Note for the production rollout: Rook does not delete a ServiceMonitor when
this flag flips, so the existing objects stay and Flux adopts them rather
than creating new ones.
The operator chart needs no change. Its own `monitoring.enabled` gates only
the RBAC that permits ServiceMonitor writes, and `csi.serviceMonitor.enabled`
is already false, so neither creates a monitoring.coreos.com object.
On a cold start only openbao-0 exists when unseal_openbao runs, so the loop
over `pods` unsealed exactly one node and returned. That deadlocks the
StatefulSet: podManagementPolicy is OrderedReady and a sealed pod never
passes its readiness probe, so openbao-1 is created only once openbao-0 is
unsealed — after this step has finished — and openbao-2 is never created at
all. The cluster settles at 1 of 3 replicas.
What makes it fatal rather than merely degraded is that the openbao Service
sets publishNotReadyAddresses, which the chart does so replicas can reach
each other to join raft before any of them is unsealed. The sealed openbao-1
is therefore a live backend on the service IP, and clients get balanced onto
a node that cannot answer. The symptom surfaces nowhere near OpenBao:
external-secrets' ClusterSecretStore flaps between "store validated" and
`dial tcp 10.105.146.11:8200: connect: no route to host`, all nine
ExternalSecrets report "could not get secret data from provider", and
infrastructure-crds — whose `wait: true` covers them — never goes Ready,
blocking all twenty app Kustomizations behind it.
The unseal step now converges instead of iterating once: re-list the pods
each round, unseal whatever is sealed, and finish when spec.replicas of them
are unsealed. A pod still in Init cannot be unsealed yet and is retried
rather than treated as a failure, which is how the OrderedReady chain is
meant to be walked.
Also drops the `2>/dev/null` on the seal-status read; stderr goes to the
same .baoerr file the wait above already uses.
infrastructure-crds stopped with
CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
namespaces "woodpecker-ci" not found
and took all twenty app Kustomizations with it, since every one of them
dependsOn it. post-crds ships that namespace's policies through
../network-policies/ and runs before any app KS; the namespace itself was
declared in apps/woodpecker, which runs after.
namespaces.yaml already explains this exact trap, two objects above, for the
sibling namespace: 'Declared here rather than alongside the workloads ...
Creating it in the woodpecker workload KS would invert that order.' It was
written for `woodpecker` and never carried across to `woodpecker-ci`.
Moving a Namespace between Kustomizations prunes and recreates it, taking its
contents with it — the hazard that ruled out splitting the rook operator
earlier. Checked before doing it: production's woodpecker-ci holds no
workloads and no PVCs, only per-step Secrets left behind by pipelines that
finished weeks ago. Losing those costs nothing.
Third instance of the same shape today, after velero/cilium and the
CRD-dependent manifests: the lesson written down beside the code, and not
applied to the case next to it.
infrastructure never went Ready on an empty cluster. Six of its seven
healthChecks were met; cilium's HelmRelease was not:
execution error at (cilium/templates/validate.yaml:79:13):
Service Monitor requires monitoring.coreos.com/v1 CRDs
Those CRDs come from kube-prometheus-stack, which lives in monitoring, which
dependsOn grafana-db, which dependsOn infrastructure. The CNI — the first thing
that has to exist — was waiting on the last.
Cilium needs none of this. prometheus.enabled is what exposes the agent metrics
the comment there argues for, and it needs no CRDs. serviceMonitor.enabled only
asks the chart to also write the scrape wiring, which is a monitoring-stack
concern expressed in the CNI's release. It moves to apps/monitoring-rules, the
layer that dependsOn monitoring and therefore has the CRDs, alongside the
ServiceMonitors for alloy, loki, immich, smartctl, velero and authentik.
The manifest reproduces cilium 1.20.1's own output field for field, read off
the live object: same name, namespace, selector, 10s interval, node relabeling.
Scraping is unchanged.
velero/helmrelease.yaml already did exactly this and explains why in a comment
that names the rebuild case precisely. It was never applied to cilium, where
the same gate is worse: velero's chart skips its template silently, cilium's
aborts the render, and cilium is a healthCheck.
Third shape, and this one is what OpenBao asks for in its own error text:
you are executing outside of a terminal (tty) ... If this is not an option,
the unseal key can be provided as the first argument
`unseal -` was Vault syntax and sent the dash as the key. Dropping the
argument makes OpenBao prompt, and the prompt wants a real terminal — from a
Job it answers 'file descriptor 0 is not a terminal'. There is no piped form.
So the key goes in as an argument and is visible in the pod's process list for
the length of the call, which the code here deliberately avoided before. That
is a real cost, taken knowingly: reading that list needs exec into the very
pod holding the key in memory, and the alternative is a port-forward and curl
against /v1/sys/unseal from outside — the openbao image carries no curl, wget
or nc — which puts a background process in the most critical step of a
recovery. Say so if you would rather pay that instead.
Run 18 reached this again with everything before it working: Ceph HEALTH_OK,
three OSDs, PVCs bound, the kopia repository read and
openbao-raft-20260902-010002.snap extracted.
Both unseal call sites ran `bao operator unseal -`, which is Vault syntax.
OpenBao's operator_unseal takes the key as argv[1] and has no case for "-":
case 1:
unsealKey = strings.TrimSpace(args[0])
so the dash itself was sent as the key and the server answered
* 'key' must be a valid hex or base64 string
Dropping the argument is what makes it read stdin, which is what the comment
beside it always claimed and what keeps the key out of the pod's process list.
Both sites fixed; nothing else in the script uses a dash that way.
Neither path had ever been reached before, so neither had ever run. The
restore got this far for the first time on run 17 and did the hard part
correctly: it read the kopia repository at Hetzner read-only, found
openbao-raft-20260902-010002.snap, pulled 839768 bytes out of it and
initialised the empty server — then could not unseal what it had just
initialised.
A rehearsal reached the restore for the first time and stopped at
FAILED: kopia not found — needed to read the Velero backup repository
without Velero
which is the one step that cannot be delegated: Velero's credentials live in
OpenBao, so Velero cannot be what restores OpenBao. bootstrap.sh reads the
kopia repository directly instead, and on a workstation kopia is simply
installed. In the Job it has to be in the image.
Added the same way as talosctl, kubectl and flux: pinned version, checksum
verified against upstream's checksums.txt, binary lifted out of the versioned
directory the tarball unpacks into.
Everything before this point now works unaided — Ceph HEALTH_OK with three
OSDs, all PVCs bound, openbao-0 running and unsealed. This is the next thing
in the way, not a regression.
One line after the previous fix, the same assumption again:
initialized=$(jq -r '.initialized // empty' <<<"$status")
jq's // is the alternative operator and treats false exactly like null, so the
one answer a cold rebuild always produces — {"initialized": false} — came out
as an empty string. The guard below then reported that OpenBao's state could
not be read at all, on a pod that had answered correctly and completely.
Presence and value are now asked separately. Verified against all three
inputs: false yields "false", true yields "true", and a genuinely missing
field still yields empty so the guard keeps working.
Run 15 got past the wait fixed in the previous commit and died here instead,
with Ceph HEALTH_OK, three OSDs, every PVC bound and openbao-0 Running. The
cluster has now come up unaided twice; both times the script threw it away one
step later. The remaining swept-for pattern is clean — the only other jq -e
tests a comparison, which is a real boolean.
The wait for openbao-0 asked jq -e '.initialized', which tests whether the
value is true. A freshly created OpenBao answers
{"initialized": false, "sealed": true}
which is complete, correct, and the only answer a rebuild can produce. jq -e
returned 1, the loop treated it as silence, and after 90 rounds against a
perfectly healthy pod the run died with 'never answered bao status'.
So the check could never pass in the one situation it exists for. It only ever
succeeded against an OpenBao that was already initialised — which on a cold
rebuild is precisely what is not there.
The comment two lines above already said an exit code of 2 is an answer and
not a failure. The code then asked the wrong question anyway. It now asks
whether the field is present, which is what 'did it answer' means; a truncated
or non-JSON reply still fails, as it should.
Found on run 14, one step after the cluster came up unaided for the first
time: Ceph HEALTH_OK, three OSDs, all PVCs bound, openbao-0 Running. The
storage chain was finally whole and this wait threw it away.
Every CSI pod in this cluster has been talking to the mons with client
libraries a whole major version ahead of them. The ceph-csi image ships Ceph
20.2.1 (Tentacle); the cluster runs 19.2.6 (Squid). Verified in both the live
cluster and a rebuild rehearsal — same images, same skew.
The image policy said the 19 -> 20 cutover should be deliberate and tied to the
Rook operator version. The coupling it protected was already gone: Rook's
floating constraint pulled v1.20 in June, which is where the v20 client came
from, and the cluster stayed on 19. The rule held while its premise quietly
did not.
It is not cosmetic. The 19.2.6 mons mint auth keys as aes256k and the 20.2.1
client cannot decode them:
auth: failed to decode key 'AgB...'
Storage works today only because the four CSI users predate the cipher change
and still carry legacy aes keys — the ones Ceph's own health check flags as
"5 auth client entities with insecure key types". Recreate any one of them, by
rotation or a deleted Secret or Rook resyncing, and the provisioner stops: no
PVC binds and nothing mounts. A cold rebuild has no legacy keys at all, so it
hits this immediately, which is how it was found. Proven both ways on the test
cluster: with aes256k every request fails, and after rotating the same user to
aes the same provisioner reports success and the volumes bind.
So the cutover happens now, to the version the client already is. Rook v1.20
supports Tentacle from v20.2.1. v20.2.0 is excluded on purpose — upstream
reports data corruption there when read affinity is enabled.
Unverified and deliberately so: this is a Ceph major upgrade against live
storage. It is on this branch to be rehearsed first, not merged.
Every PVC failed with
failed to get connection: connecting failed: rados: ret=-22, Invalid argument
and it was mine. The Driver CRs on the live cluster carry clusterName:
rook-ceph; I copied the driver names out of them and not this field, and the
chart defaults it to the empty string. It becomes --clustername on the plugin
and goes to rados_create2 as the Ceph cluster name, which librados refuses.
The failure is a good imitation of a credential problem. Ruled out, each
against the live cluster: the CSI user exists and is named identically in both
clusters, the key in the Secret hashes equal to the one Ceph holds, that same
user and key connect fine from the toolbox, monitors in ceph-csi-config match
the mon Services exactly, ClientProfile is byte-identical to production, and
the images are the same cephcsi v3.17.0. A detour through Ceph key ciphers
looked convincing — production's CSI users still carry the legacy aes type
while a fresh cluster mints aes256k — and was wrong: the aes256k key
authenticates when passed the way ceph-csi passes it.
What gave it away was diffing the plugin's own arguments against production's,
which is where the missing flag is plainly visible.
A cold rebuild never gets storage. Rook stopped deploying CSI in v1.20 and
hands it to the ceph-csi-operator, which does nothing until someone supplies
Driver CRs. On this cluster Rook's v1.18 -> v1.20 migration created them once,
116 days ago, so they look like part of the furniture; on an empty cluster Rook
creates the CephConnection and the ClientProfile and stops. No Driver, no
provisioner, and every PVC waits forever — OpenBao's included, which is exactly
where the rehearsal stopped, three times, after Ceph itself was healthy with
three OSDs.
The workaround this replaces (babed078) rendered the chart's ServiceAccounts and
RBAC as static manifests and left the CRs to the migration. It was written
during a five-day outage with PVC attach blocked cluster-wide, and it gave a
reason that does not hold: helm-controller v1.5.4 does not "predate
disableTakeOwnership". That landed in v1.2.0, three minors earlier, and it is an
opt-out — taking ownership of existing resources is the default. The chart was
installable all along.
cephConnections and clientProfiles are left empty deliberately. Rook owns both:
the CephConnection carries mon Service IPs Rook keeps current, and the
ClientProfile references Secrets Rook creates. Helm managing either would freeze
a value meant to move. Confirmed against a fresh cluster, where Rook creates
precisely those two and no Driver.
The driver names are pinned to rook-ceph.{rbd,cephfs}.csi.ceph.com, matching
what the StorageClasses already name as their provisioner. The chart's default
omits the prefix, which would leave every StorageClass pointing at nothing.
On a cluster that already has these objects, Helm adopts them. That has not been
exercised yet — this branch is not for master until it has.
With the memory fix in place the workers moved from 99% memory requests to
49%, and straight into the next wall at 69% of CPU. What could not be
scheduled this time was rook-ceph-operator itself, plus the multus
DaemonSet's new pods during a rollout.
An unscheduled operator is a cluster with nothing driving it: two OSDs
crash-looped with no one to reconcile them, the MDS never placed, and the
PVCs sat on 'Waiting for a volume to be created by the external provisioner'
for half an hour because the provisioner's operator was Pending. Ceph
reported Ready throughout, which is true of the CR and says nothing about
whether storage works.
Workers get 4 CPUs and the pod requests 6. k8s-3 is at 24% of its CPU
requests, so this is not close to anything.
Ceph came up with three OSDs and then went HEALTH_ERR. The OSDs crash-looped
on
unable to find any IPv4 address in networks '10.99.0.0/29' interfaces ''
Failed to pick cluster address.
which reads like an addressing fault and is not one. The whereabouts pool
had zero allocations and the OSD pods carried the ceph-san request
annotation with no network-status to match it, because kube-multus-ds and
whereabouts were themselves Pending: a worker sat at 99% of its memory
requests once it carried an OSD, a mon and a mgr, and there was no room left
for the CNI DaemonSets. The MDS pods could not schedule either, which is
where the HEALTH_ERR came from.
Workers go to 8GiB and the pod ceiling to 36Gi. k8s-3 carries it at about
half its allocatable memory.
The alternative was shrinking Ceph's requests until they fit, which would
have made the test pass by measuring something production does not run.
Ceph reached Ready with zero OSDs. All three osd-prepare jobs completed
cleanly and refused the disk they were pointed at:
skipping device "vdb1": ["Used by ceph-disk"]
skipping OSD configuration as no devices matched the storage settings
Rook classifies a partition as belonging to an old ceph-disk deployment by
its label, and Talos names a RawVolumeConfig's partition r-<name> — so
'ceph-osd' became 'r-ceph-osd' and Rook read its own name in it and backed
off. The partition type GUID is the generic Linux one, so the label is the
only thing it could have matched on.
Renamed to osd-data, giving r-osd-data. Nothing about the volume changes but
the string. Production is unaffected either way: there the OSD is a whole
NVMe matched by device id, not a partition, which is why this never showed
up until a VM ran the same CephCluster.
Ceph itself now comes up — mons, both mgrs, mds, and the network canary all
pass, which is what the last three commits were about. This is the next link
after that: still no storage, so OpenBao's PVC still cannot bind.
The NetworkAttachmentDefinition set mtu 9000, and so does the LinkConfig on
enp2s0f1np1 that it attaches to. Two places for one number, which have to
agree and are not checked against each other.
They disagreed on the rebuild rehearsal and the attachment was refused
outright: 'invalid MTU 9000, must be [0, master MTU(1500)]'. A VM's NIC is
capped at 1500 by the host_mtu qemu advertises, and the virtio driver will
not let the guest exceed it, so jumbo frames are not reachable there at all.
The plugin's own error names the fix — 0, meaning unset, is a valid value
and inherits from the master. Production behaviour is unchanged, because the
master already carries 9000; the number is now stated once, on the link that
actually has to carry the frames.
Third error in the same chain, each one a layer further in: the plugin was
missing (mirrored), then the link was missing (VLAN), now the MTU. Ceph is
still not up.
Ceph never came up in the rehearsal. Rook's network canary could not attach
ceph-san and the CephCluster sat at 'Configuring Ceph Mons', so ceph-block
never existed, so OpenBao's PVC never bound. The canary's error moved from
'failed to find plugin macvlan' to 'Link not found' once the CNI plugins
were mirrored: macvlan ran and found no enp2s0f1np1, which is a
bus-path-derived name from an X710 card that no VM will ever produce.
Production has a second physical port for that link. A VM has one NIC, so
here it is a VLAN on it, deliberately carrying the same name — the test then
runs san-network-attachment.yaml verbatim rather than a patched copy, which
is the whole point of the exercise.
Two dead ends first, both found on throwaway nodes rather than in
production. LinkAliasConfig looks like the right tool and is not: Talos
records the alias in its own field, not in altNames, so macvlan — which
resolves its master by kernel link name — cannot see it. And a VLANConfig on
its own took vm-4 off the network entirely: declaring any network document
makes Talos configure the link explicitly and the implicit DHCP that
addresses the node stops. The DHCPv4Config restores it, verified on a second
node before being written down.
A dummy link would have been simpler and worthless: no traffic between
nodes, so Ceph would replicate over an interface going nowhere and the test
would go green having proven nothing.
This is machine-level config, not site-level. The link belongs to the
machine that has it.
A cold-rebuild rehearsal stopped dead here. The installer DaemonSet pulls
gitea.renner.uno/gitea_admin/cni-plugins, and Gitea runs inside the cluster
being rebuilt, so all four nodes sat in ImagePullBackOff with 'no route to
host'. Without the plugins there is no macvlan; without macvlan Multus
cannot attach ceph-san; without that network Rook's canary never starts,
the CephCluster hangs at 'Configuring Ceph Mons', and OpenBao waits forever
for a PVC that can never bind. Four layers between the cause and the symptom.
Baking the binaries into an image fixed a real integrity gap — the old path
fetched a release tarball on the node with no checksum at all. What it also
did, unnoticed, was trade github.com on the critical path for something
strictly worse: a registry that by definition does not exist during a
recovery. The mirror keeps the checksum and removes the dependency.
ghcr.io/konairius/cni-plugins:2026-07-30 is the same image, verified by
digest (sha256:85e83e32...) against the Gitea copy, and public: an
unauthenticated pull was confirmed to work. It has to stay credential-free,
because an imagePullSecret would come from OpenBao, which needs Ceph, which
needs these binaries.
Every other image in this directory already came from a public registry.
The build pipeline does not yet publish to ghcr.io — that needs a token this
commit cannot provide, so today's mirror was pushed by hand. Bumping the tag
means mirroring first until that lands.
At 10GiB the system disk yielded an 8.36GB /var, which the infrastructure
image set filled to 82%. kubelet then tainted all four nodes with
node.kubernetes.io/disk-pressure and nothing could schedule at all —
Ceph's own monitor sat Pending, and the symptom surfaced three layers
away as a Cilium HelmRelease stuck InProgress, which is not where anyone
would look. Now 40GiB.
The second disk is the Ceph OSD. Three 8GiB OSDs replicated 3x leave 8GiB
usable, less than OpenBao's two PVCs ask for on their own, so the storage
could not have satisfied the cluster it was being tested with. Now 24GiB.
Both are raw sparse files: the larger geometry costs only what is written.
The emptyDir ceiling moves to 150Gi, which still bounds actual usage well
below the ~810GB free on the filesystem behind it.
infrastructure-ceph-cluster no longer dependsOn infrastructure. That single edge
closed a loop, found by rebuilding from an empty API server:
infrastructure is Ready only once its openbao healthCheck passes
-> the openbao HelmRelease is Ready only once openbao-0 runs
-> openbao-0 stays Pending on "unbound immediate PersistentVolumeClaims"
-> data-openbao-0 wants StorageClass ceph-block
-> ceph-block is defined in infrastructure/rook-ceph-cluster/
-> applied by infrastructure-ceph-cluster, which waited on infrastructure
Nothing in that chain can move. Every StorageClass in the cluster is Ceph-backed,
so there is no storage OpenBao could use that exists before Ceph, and the loop
cannot be broken from the storage side at all.
The edge was never load-bearing. infrastructure runs wait: false, so it applies
the rook-ceph namespace and the operator the moment it reconciles; only its
readiness waits on the healthChecks. Verified on the deadlocked test cluster:
removing the dependsOn created the CephCluster and CephBlockPool within seconds
and ceph-block appeared, with infrastructure still not Ready. On an empty API
server this Kustomization now retries until the operator installs its CRDs — the
shape post-crds already relies on — and on a live cluster nothing changes,
because the CRDs are always there.
The alternative was dropping the openbao healthCheck, which breaks the cycle
just as well. It was rejected: the guarantee it encodes belongs somewhere, and
splitting the rook operator into its own Kustomization was rejected too, because
it would move a Namespace between Kustomizations and a prune on a Namespace
takes everything inside it.
Also moves authentik's outpost ServiceMonitors to apps/monitoring-rules/. Their
CRDs come from kube-prometheus-stack, and the authentik Kustomization depends on
infrastructure* but not on monitoring — so on a cold cluster it raced the chart
and applied nothing until it happened to lose. Unlike the Ceph loop this one
resolves itself, but authentik is the login provider a lot waits behind. No
ServiceMonitor, PodMonitor, PrometheusRule or Probe now sits outside the layer
whose dependency installs its CRD.
The whole script has run in the pod since the flux and OpenBao half moved
there, but the comment still told a reader that half stays on the
workstation, and still quoted the old 2.5G control plane.
Two harness defects, both found by a run rather than by reading.
The control plane ran on 2560MiB. Measured on the live VM mid-run: 250MiB
available, page cache squeezed to 386MiB, 94MiB free. etcd's fsyncs therefore
landed on a system in permanent reclaim, and linearized reads took 5 to 20
seconds against an expected 100ms. The apiserver became unreachable, recovered,
went again; the run sat in a bounded wait printing nothing, because its bound
counts iterations and each iteration was blocking on API timeouts. Nothing in
that failure says whether the repo can rebuild a cluster, which is the only
question this test exists to answer, so the control plane gets 6GiB and the pod
ceiling moves to 24Gi. k8s-3 carries it comfortably: requests were at 30% of
93GiB allocatable.
The image is pinned by digest. Spegel mirrors mutable tags and keeps serving
what it cached, so a run on k8s-3 pulled a ci-vmtest:latest from three builds
back — one that predated OVMF — and died with "the required flash image was not
found" before the first VM booted. The registry's :latest was a different digest
at that moment, which is what made the diagnosis unambiguous. The dated tag rides
along for readability; containerd resolves the digest.
The wait for OpenBao is supposed to end on one of two cluster facts: the
StatefulSet appears, or Flux reports the infrastructure Kustomization has failed.
Only the first ever fired. The second read READY as the fourth field of
`flux get kustomization --no-header`, whose columns are NAME REVISION SUSPENDED
READY MESSAGE — but REVISION is empty until the first successful reconcile, which
is exactly the state this check exists to catch. The columns shift left, the
fourth field lands on the message, and matching it against "False" never fires.
So the persistent-failure counter never moved. Yesterday's run sat for 45 minutes
on an infrastructure Kustomization that had been failing its dry-run since its
first minute, and would have run the full 720 iterations — two hours — before
saying anything. The bound meant as a backstop was doing all the work, which is
the failure this wait was written to avoid.
The Ready condition has no positional ambiguity, so all three waits now read it
directly via kubectl. ks_ready_condition echoes "<status>|<message>"; an empty
status means the object does not exist yet, and kubectl's own error is carried
through as the message rather than discarded, keeping "not created yet"
distinguishable from "the lookup itself broke". Verified against the live cluster
for both the found and not-found paths.
The HelmRelease tally in the summary gains a denominator. "12 ready" reads as
success whether the cluster has 12 HelmReleases or 40, and a rebuild is precisely
where that distinction matters. Both numbers now come from the objects.
A cold-rebuild test against an empty API server stopped here permanently:
infrastructure False ExternalSecret/apps/external-dns-unifi-secret dry-run
failed: no matches for kind "ExternalSecret" in version "external-secrets.io/v1"
Flux server-side dry-runs the whole apply set, so a single unknown kind rejects
all of it — including the ESO HelmRelease that installs the very CRD the dry-run
is missing. The VM cluster confirmed it: zero HelmReleases, zero ESO CRDs. The
deadlock cannot resolve itself, and no amount of waiting changes that.
Eight ExternalSecrets and three Traefik Middlewares sat beside their consumers
under infrastructure/, whose own HelmReleases install their CRDs. They move to
post-crds/, which exists for exactly this and already held the OpenBao
ClusterSecretStore and the Velero Schedules on the same grounds.
Velero's ServiceMonitor and PodMonitor could never have applied there at all:
monitoring.coreos.com CRDs come from kube-prometheus-stack under apps/, which
reconciles after infrastructure. They move to apps/monitoring-rules/, whose Flux
Kustomization already carries a comment describing this precise failure for
PrometheusRule — the same diagnosis, made once and never carried across.
None of this was reachable in production, where the CRDs have existed since the
day each operator was introduced. Ordering only becomes load-bearing when
something applies it against nothing, and until now nothing ever had.
The ExternalSecret in openbao/rotation/flux-git.yaml is split out; the
ServiceAccount and rotation CronJob stay where they are.
Merging this transfers ownership between Kustomizations, so Flux prunes each
object from infrastructure and recreates it under infrastructure-crds or
monitoring-rules. The Secrets ESO owns vanish for that window — running pods
keep their mounted copies, but a pod starting in the gap will not find them.
Found by reviewing rather than by running, after several twenty-minute runs each
surfaced one more. Every one of these was introduced today.
The documented cold-rebuild command could not work. require_secrets resolved the
git token from FLUX_TOKEN — a name that appears nowhere else in the repo, while
the usage header and RUNBOOK-cold-rebuild tell the operator to export
GITHUB_TOKEN or GITEA_TOKEN. Worse than a typo: `${VAR:?}` inside eval terminates
the shell outright, so `FLUX_SOURCE=github GITHUB_TOKEN=… ./bootstrap.sh flux`
printed "Collecting secrets... " and exited, with no message, no diagnosis, and
no cleanup — the EXIT trap does not run either. It now reads the variables the
documentation names.
A second `trap ... EXIT` replaces the first rather than chaining, and the
secrets-cleanup handler was registered after the vmtest teardown. A workstation
rehearsal therefore left four VMs, their bridges and the sudo keepalive running,
and ignored VMTEST_KEEP — exactly the "next run collides with the last" failure
the teardown exists to prevent, and only on the laptop path. There is now one
handler, and the others register with it.
wait_kustomization and wait_certificates returned 1 on timeout, called bare under
`set -e`. They printed "continuing anyway" and then ended the run — every wait
after the first slow app never ran, and the result read as "bootstrap broke at
apps" when apps was merely slower than a guessed bound.
The `grep -c Ready` bug survived in the verification block, where it was reported
as fixed elsewhere: NotReady contains Ready, so a cluster with no ready node
printed 3/3. `grep -c True` on Kustomizations and HelmReleases matched the
SUSPENDED column, so a Kustomization Flux is not reconciling read as ready. Both
now match the field whole, verified against live output — READY is column 4 for
Kustomizations and 5 for HelmReleases with -A. awk also avoids grep's exit 1 on
zero matches, which under pipefail had ended the run with no output.
The in-cluster run failed with the VM cluster unable to reach Gitea at
10.2.99.86 — and that is not a fault in the test environment. Gitea runs inside
the cluster being rebuilt, so in a real cold rebuild it is exactly as
unreachable. The test reproduced the disaster condition faithfully; the answer is
the one already in this script, FLUX_SOURCE=github.
Worth recording that the first diagnosis was wrong: nested NAT and a VXLAN MTU
looked like the cause, and the provisioner has no MTU flag, so the next step was
going to be measuring inside the VMs. The reachability was never the problem.
The secrets list demanded a Gitea token regardless of FLUX_SOURCE, so a run
rehearsing the break-glass path could not have obtained its token — the rehearsal
was wired to need the one thing a disaster removes. One entry now covers both and
is handed to whichever variable the chosen source reads.
The recovery path must not be the one that only runs during a recovery. Secrets
were read where they were needed — from pass off-cluster, from files in the Job —
and the upfront check existed only in the Job. So a missing key surfaced an hour
into a run, and the laptop branch, the one that matters most, had no check at
all.
One function now collects all of them into one directory before anything else
happens: on a laptop by running the commands the list names, in the Job by
waiting for the operator to stream them in. Everything downstream reads files and
cannot tell the two apart, which is the point — the difference between a
rehearsal and a real recovery is confined to those few lines.
The paths live in that list and nowhere else. They were also sitting in
OPENBAO_PASS_* constants, which is how one value ends up defined twice and a copy
goes stale — as had already happened here with a gitea pass path that does not
exist.
Verified on the workstation: all five collected, the directory is 700, and the
trap removes it on exit.
The test wants about 14.5G — one control plane at 2.5G and three OSD workers at
4G. The workstation has 31G with a desktop on it: a single run left under 1G
free, and when a killed run's teardown could not get sudo and the next run
started beside it, the OOM killer took the session. The nodes have ~94G each
with roughly 70G unused, and /dev/kvm is present on them.
Three pieces:
ci/images/vmtest carries what the phase shells out to — qemu, talosctl, kubectl,
flux, helm, yq, jq. Separate from ci-k8s rather than built on it, because that
image pins talosctl v1.12.6 while this creates a cluster at the version
cluster.yaml installs; a client older than the Talos it provisions is the kind of
skew this test exists to catch.
kubernetes/vmtest-job.yaml runs it. Deliberately outside kubernetes/clusters and
so not reconciled by Flux — a Kustomization would recreate a one-shot Job after
every completion.
The phase learned that root is not always sudo: in the Job the process is already
root and sudo is not installed, so the elevation became a variable. The
keep-alive that stops sudo's timestamp expiring mid-run applies only off-cluster.
Two things are deliberate rather than incidental:
The pod is privileged. It needs /dev/kvm and it builds bridges and taps. It does
NOT need hostNetwork — the provisioner's network lives in the pod's own
namespace and the script talking to 10.5.0.x runs in the same pod — and it does
not mount a service account token, so a stray kubectl cannot reach the cluster it
is running in.
The memory limit is not decoration. The failure it guards against has already
happened once, and on a node it would take real workloads rather than a session.
What stays on the workstation is the flux and OpenBao half. That needs the unseal
key from the operator's pass store, and putting the key that protects OpenBao
into a pod inside the cluster OpenBao secures gives up the reason it lives
outside. It costs almost no memory, which is the half the workstation can afford.
The documented cold rebuild could not work. Velero's bucket credentials come from
an ExternalSecret, which needs the ClusterSecretStore, which needs an unsealed
OpenBao — so Velero cannot be the thing that restores OpenBao. That circle is why
the runbook falls back to a human with `kubectl cp`, and why the migration doc
still says "Still owed: one tested restore".
Nothing new is produced to break it. The snapshots are already offsite: the
nightly CronJob writes them to a PVC and the openbao-daily Schedule ships that
PVC through Velero's data mover into a kopia repository. Verified from outside
the cluster — seven daily snapshots, newest 839KB from this morning.
The phase reads them directly with kopia. All three things it needs live outside
the cluster, which is the whole point: bucket credentials from pass (their own
key, not Velero's), the kopia binary, and the repository password — which is not
a secret, Velero hardcodes `static-passw0rd` in pkg/repository/keys/keys.go.
Worth stating plainly: the offsite backup's confidentiality therefore rests on
bucket access control alone.
The restore sequence follows from how snapshots and seal keys interact. A
snapshot can only be loaded into a running server, so the empty one is
initialised and unsealed with a throwaway key that exists for a few seconds and
is never stored. `-force` is then required, because OpenBao otherwise refuses a
snapshot whose keyring does not match the cluster's own — which is exactly the
case here. Restoring replaces the keyring, leaving the server sealed against the
original key, and the existing unseal step carries on from there with the key
from pass.
Guarded on the one case that destroys data: this runs only when OpenBao reports
initialized: false. Against an initialised server nothing happens. And with no
unseal key in pass it stops rather than initialising a fresh OpenBao, because
that is a decision and not a step.
Adds kopia and jq as requirements, both checked with a message rather than
assumed.
Never executed yet — the run this enables is the test.
A cold rebuild could not reach a reconciled state without someone at a keyboard.
infrastructure-crds waits on the ClusterSecretStore, that needs an unsealed
OpenBao, and every app Kustomization waits on infrastructure-crds — so eight
typed commands sat in the middle of the disaster path. The VM test made this
concrete: everything downstream simply stops there.
The flux phase now unseals every replica before it starts waiting on
infrastructure. The unseal key stays exactly where it was, in the operator's
`pass` store, and is read at run time; it is never written to the repo and never
reaches CI, which is why this was left manual to begin with. It goes to the pod
on stdin rather than as an argument, so it does not appear in a process list.
Idempotent by design: a replica that is already unsealed is skipped, so this is
safe inside `bootstrap.sh all` rather than being a one-shot.
What it deliberately does NOT do is initialise. `bao status` distinguishes sealed
from uninitialised, and the two are different situations: a rebuild restores a
raft snapshot and reuses the existing key, a genuinely new cluster runs `bao
operator init`. Initialising over an OpenBao that already holds data is how the
data is lost, so an uninitialised server stops the run with a message naming
which of the two applies. Restoring the snapshot is still a human decision and
still manual.
Also clears a cluster left behind by a killed run before creating a new one. A
run that is killed never reaches its EXIT trap, and the leftover VMs keep the
provisioner's CIDR — the next start then failed with what looked like a
networking fault.
The phase stopped after the Talos phase, which is the part that needs nothing but
the machines and can therefore run unattended. VMTEST_FULL=1 continues into flux
and wait-apps instead — the same code a real rebuild runs, against VMs.
It is opt-in because it needs two things the Talos half does not: GITEA_TOKEN,
and a branch for `flux bootstrap` to commit its sync manifests into. That branch
has to exist already and must not be master, so the phase refuses to start
otherwise rather than discovering it after the VMs are up.
Nothing else changes: the phase still sets the site variables and falls through
into the real phases rather than reimplementing them, and the teardown still runs
whatever happens.
2026-08-29 11:21:36 +02:00
63 changed files with 3010 additions and 2412 deletions
# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
# The default is unchanged and deliberately so: "secrets" resolves to working
# credentials and this script has always used it. Both "secrets" and "hetzner"
# exist in ~/.aws/credentials on the workstation.
#
# The defect is only the assignment. Forcing the value defeated the env-based
# path this file's own header offers, and it also overrode a caller who had
# already selected a profile — which is how a 403 from HeadObject can appear
# while the credentials the caller intended were perfectly good. That error
# reads as a bucket permissions problem and sends you looking in the wrong
# place.
exportAWS_PROFILE="${AWS_PROFILE:-secrets}"
command -v aws >/dev/null ||{echo"aws-cli not found";exit 1;}
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner