The rewrite earlier today stated, at the top and again in known gaps, that the
rebuild cannot be driven remotely: management is a tagged VLAN, a maintenance
mode node cannot configure VLANs, so it comes up unreachable and every machine
needs a keyboard.
That is wrong. A maintenance-mode node DHCPs untagged and is reachable at that
address — it is not on management, but it answers, and that is precisely the
address `apply-config --insecure` targets in Step 2. All three bare-metal
conversions were driven that way.
The claim came from #576 and was carried into this document without being
checked against what had actually happened days earlier in this repo. It is the
worst place to put an unverified constraint: a disaster-recovery runbook is read
when there is no capacity to discover that a stated prerequisite is fiction, and
"you need physical access to three machines" is exactly the kind of statement
that changes whether someone attempts a recovery at all.
What survives is narrower and true: MS-01 has no IPMI and nothing here does
PXE, so booting from USB is physical. Everything after the boot is remote.
Also removes the follow-on error from the first correction, which claimed the
switch port change on 2026-08-27 had fixed a reachability problem. There was no
reachability problem to fix. That change is real and useful — a maintenance mode
node now lands in management rather than the fallback network — but it did not
turn an impossible rebuild into a possible one.
#576 still carries the original claim and is flagged in known gaps as wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 was "tofu: provision the VMs" and there are no VMs. The OpenTofu stack
has no node definitions left — only the uninstantiated proxmox_vm module beside
DNS, object storage and UniFi — so `tofu apply` provisions nothing and Step 2
then has no cluster to form. The document was half-updated: it knew k8s-1 was
bare metal but still treated k8s-2 and k8s-3 as Proxmox VMs, which stopped
being true at the neptune conversion.
That matters more than an ordinary stale doc, because it is the document
someone reaches for when everything is gone — exactly when there is no capacity
to discover that step one is fiction.
Step 1 splits into two unrelated halves: a tofu apply for platform resources
that do still exist, and physically booting three machines from install media.
Step 2 stops calling `bootstrap.sh talos`, which cannot generate a config
(#581), and carries the procedure that actually works — the one
RUNBOOK-baremetal-conversion.md uses. Both traps it hides are now stated: the
whole cilium-cni.yaml cannot be passed as a patch, and the generated
HostnameConfig collides with the per-node hostname. So is the two-stage apply:
a config referencing an interface that does not exist yet hangs the install at
STAGE: Installing with nothing timing out.
Scoped that claim rather than repeating "bootstrap.sh is dead" wholesale — the
flux and wait-apps phases are still used in Steps 3, 5 and 6 and work. Only
config generation is broken, and saying otherwise would have left the runbook
contradicting itself three sections later.
Adds the constraint that decides whether this is even attemptable remotely, at
the top and again in known gaps: management is a tagged VLAN, a maintenance
mode node cannot configure VLANs, and MS-01 has no IPMI. Every node needs a
keyboard and a monitor until #576 puts VLAN 40 native on the I226-V port. That
was true before today and undocumented.
Also records what changed under the runbook recently: the schematic now carries
i915, Ceph's cluster network is a Multus attachment rather than host
networking, and OpenBao runs three replicas that each need unsealing
individually — a sealed server is neither a voter nor a standby, so doing one
leaves the cluster short of quorum.
Disk verification before install is now explicit. Talos enumerates NVMe in an
order that is not stable across boots, the per-node configs select by serial for
that reason, and the failure mode is the installer taking the Ceph OSD drive.
Closes#660.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At 30h a single missed cycle pages critical, and a single missed cycle is the
normal cost of touching the nodes. Velero moves data through pods, so draining
a node kills whatever upload is in flight and the schedule is lost for that day
— it is not retried. Backup windows run 03:15 to 08:45 UTC, so any rolling node
work overlapping them produced a guaranteed critical alert per affected
schedule.
That is noise about planned work, not signal about backups. It fired for eleven
schedules during the bare-metal Talos upgrade and again today.
60h still catches what the rule exists for. Two consecutive failures is no
longer explicable by one maintenance window. Nothing is hidden in the meantime:
VeleroBackupFailed fires on the first failure regardless, so the first cycle is
still visible — only the escalation to critical waits for a second cycle to
confirm it.
Calibration against the current state: the four stale schedules sit at 46.8 to
48.5 hours since their last success, so the new threshold leaves roughly twelve
hours — exactly until today's runs. If those fail too, the alert fires, which
is the intended behaviour rather than a gap.
Verified: the rule passes a server-side dry run, the expression returns nothing
at 60h, and the same query at a lower bound returns the four affected schedules
with their ages — so the selector is known to match rather than merely silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes run v1.12.11, which is the last 1.12 patch — v1.12.12 does not exist.
v1.13.9 is current.
The upgrade path is the documented one rather than a shortcut. Migration is
only tested between adjacent minor releases, and the recommendation is to go
to the latest patch of each intermediate minor. 1.12.11 -> 1.13.9 satisfies
both: one minor step, taken from the last patch of the version being left. No
intermediate hop is needed.
Kubernetes is not forced along. Talos 1.13's support matrix lists 1.36, 1.35,
1.34, 1.33, 1.32 and 1.31, and the cluster runs 1.35.2 — so it stays there
until `talosctl upgrade-k8s` moves it as a separate decision. This was the one
open question before the jump and it is answered from the matrix rather than
assumed.
Deprecations in 1.13 were checked against this cluster, not just read:
.machine.env -> EnvironmentConfig not used
.machine.network.kubespan -> KubeSpanConfig not used
EtcdConfigs/KubeletConfigs protobuf change no consumer in this repo
legacy upgrade API -> LifecycleService still functional; `talosctl
upgrade` keeps working, this
only matters once it is removed
Both the five machine-config files here and the live config on k8s-1 were
checked for the first two, because the running config is generated from a base
this repo only patches and could carry fields the patches never mention.
Neither appears in either.
The removed interactive installer and the dropped `talosctl install` do not
apply — installation here is `talosctl upgrade --image` and Image Factory
media.
Schematic is unchanged; the factory image was verified to exist for the new
tag before committing. The cold-rebuild ISO URL moves with it.
Not applied to any node. Rolling it out is drain -> upgrade -> wait for Ceph
to return to 217 PGs active+clean, one node at a time, as in the 1.12.11 run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).
Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.
Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.
mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.
kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.
Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was .4-.6 because the nodes held .1-.3 for as long as Ceph needed
host networking to reach VLAN 49 — three addresses for three OSDs, with no
spare, during the one phase where things were actually moving.
The nodes have since given those addresses up: the link is only a macvlan
parent now and does not need one of its own. The reservation has nothing left
to avoid, so the pool takes .1 through .6.
Six for three is the headroom the migration could not afford. It leaves room
for a leaked allocation to sit until the whereabouts reconciler collects it —
the case where a node is lost hard enough that the CNI DEL never runs — and
room to add a second OSD per node without touching this file.
Keeping the pool inside the /29 rather than moving it somewhere roomier on the
segment stays deliberate. There is no router here, so an address outside the
/29 is unreachable from anything using one inside it, which is what would
break a future rolling change where migrated and un-migrated OSDs still have
to talk to each other. That was nearly the shape of this attachment and it
would have taken every pool down at the first OSD.
Inert for the running cluster: a NetworkAttachmentDefinition is read when a
pod is created, so the three existing OSDs keep .4, .5 and .6 until they next
restart for some other reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all
while IPv4 from the same pod worked normally.
enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0
is unused. Both sit on the untagged fallback network, where by design nothing
is supposed to run — and both were autoconfiguring from its router
advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.
That left four IPv6 default routes per node at identical priority 1024 —
mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally
routable source address. With nothing to break the tie, outbound IPv6 mostly
left through an interface whose only usable source was a ULA.
This is the same argument the file already makes for ens19, where host
addresses on a link that exists only as a macvlan parent are called "never
needed and actively harmful". It was simply never extended to the physical
ports.
How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had
been firing for 30 hours, reading as though the proxies were down. They were
not — every failing target is an IPv6 literal and the blackbox exporter is a
pod, so the probes were failing on cluster egress. The proxies answered IPv6
in 13 ms from a workstation on another VLAN throughout.
Applied to k8s-1 first and verified before rolling to the other two, because
the existing precedent disables v6 on a VLAN *child* while this targets VLAN
*parents*, and mgmt0 carries the address Ceph and Cilium use:
mgmt0 kept 2003:d6:9718:3b02:… and fd40::… unchanged
fd00:: addresses 2 -> 0
IPv6 default routes 4 -> 2 (mgmt0, lo)
failing blackbox probes 5 -> 0
Ceph 217 PGs active+clean, no OSD restart
All three nodes are already patched live; this puts it in git so the next
apply-config does not undo it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no OpenBao alerting of any kind — no rule, no ServiceMonitor. That
is why today's seal went unnoticed for two hours: the k8s-1 Talos upgrade
sealed the single replica, the ClusterSecretStore went InvalidProviderConfig,
and nothing said so.
Three replicas make this more important rather than less. Before, a seal was a
visible outage that announced itself. Now the service survives one, so a
sealed replica is a quiet loss of redundancy that nothing surfaces until the
next drain takes quorum with it.
Watches pod readiness rather than a seal metric, because there is no seal
metric to watch. OpenBao is not scraped: no ServiceMonitor exists, and its
metrics endpoint needs a token or an explicit unauthenticated_metrics_access
telemetry setting, neither configured. Verified against the running
Prometheus — no bao_* or vault_* series at all. External Secrets exports
nothing usable either, so the ClusterSecretStore going NotReady is invisible.
Readiness carries the signal for a structural reason, not by luck: a sealed
server fails its readiness probe because it can neither serve requests nor act
as a raft standby. Sealed is therefore always not-Ready. The converse does not
hold — a crash or stuck scheduling fires the same alert — but those want a
human too.
Three rules:
OpenBaoReplicaNotReady < 3 ready for 10m, warning. The duration only
ignores ordinary rescheduling; a sealed replica
never recovers on its own.
OpenBaoQuorumLost < 2 ready for 5m, critical. Raft quorum of three
is two, so below that the store is down.
OpenBaoMetricsMissing absent() for 15m, warning. Both rules above are
threshold comparisons, so if the series disappears
they stop evaluating and the silence is
indistinguishable from health.
Verified against the running Prometheus rather than assumed: the selector
matches (the same query with `< 4` returns the series at 3), and the alerting
thresholds correctly return nothing at 3/3 ready. Testing only that it returns
nothing would not have caught a typo in the selector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenBao ran a single replica. Every node drain took the whole secret store
down with it: today the k8s-1 Talos upgrade sealed it, the ClusterSecretStore
went InvalidProviderConfig, and it stayed that way for two hours because
nothing was watching.
Three, not two. Raft needs a quorum and the quorum of two is two, so a
two-replica cluster tolerates no failures — it would stop the moment one pod
is sealed or rescheduled, which is worse than the single replica it replaced.
Three tolerates one.
What this buys: a drain seals the replica on that node, and raft elects a new
leader from the remaining standbys within seconds. The service does not go
down.
What it does not buy, stated plainly because it is easy to assume otherwise: a
sealed server can neither serve requests nor act as a standby, so that replica
stays out of the cluster until someone unseals it. Scaling removes the outage,
not the manual unseal. For rolling node maintenance the consequence matters —
a Talos upgrade drains all three nodes in turn, so unseal each replica as it
comes back, or the third drain takes the last unsealed one and quorum is lost
anyway.
retry_join is what makes a scaled replica join the existing cluster rather
than come up isolated and uninitialized; without it each new pod would need a
manual `bao operator raft join`. All three peers are listed on every node —
the stanza is a list of possible leaders, a node ignores its own entry, and
listing all three keeps the config identical across replicas regardless of
which pod currently leads. Addressed through the headless service so each
entry resolves to one specific pod.
The snapshot job moves from the plain service to openbao-active. With one
replica the plain service could only ever hit the one pod; with three it
selects any of them, and a raft snapshot has to be taken against the leader.
No affinity was added: the chart already applies a required podAntiAffinity on
kubernetes.io/hostname, so the replicas spread across the three nodes on their
own — verified against the running StatefulSet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodes held 10.99.0.1-.3/29 on enp2s0f1np1 only because reaching that link
required CephCluster.spec.network.provider: host, which is not per-daemon and
put every Ceph daemon on host networking. Rook now attaches the link to the
OSD pods through a Multus macvlan, and a macvlan parent only needs to be up —
it does not need an address of its own for its children to work.
By the rule this segment is built on — Ceph-internal traffic and nothing else,
no router, no other participants — the node has no business holding an address
there either. This is stricter than before, not looser.
The link keeps up: true and mtu: 9000. Both still matter: the parent must be
up for the macvlan children to exist, and the MTU is what the children
inherit.
Not in this commit, deliberately: widening the attachment's whereabouts pool
from .4-.6 to the full /29. That must not land before these addresses are
actually gone from the running nodes, or whereabouts could hand .1 to an OSD
while a node still holds it. Follow-up once these configs are applied and
verified.
Also records the one-time step that the manifests cannot express. Removing
addressRanges did not clear public_network and cluster_network from Ceph's
config database — the same trap the bluestore comment in the same file
describes. The first OSD of the phase A rollout came up on the pod network,
still had public_network 10.2.40.0/24 in the config database, found no
interface in it and crash-looped on "Failed to pick public address". Both keys
were removed by hand. A rebuild from scratch will not hit this because the
config database starts empty; another provider change on a live cluster will.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting
every Ceph daemon on host networking to get there.
The cluster network is an OSD concern. Per the Ceph documentation it carries
OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on
the public network". Reaching it through addressRanges required
provider: host, which is not per-daemon — the mons went onto host networking
as a side effect, making a mon's address the node's DHCP lease.
That failure is documented rather than hypothetical. When a lease moved, the
mon could not bind ("Cannot assign requested address") and crash-looped. Rook
detected it and began a failover, but with three mons on three nodes each
holding the mon host ports the replacement could not be scheduled anywhere —
the canary sat Pending on "didn't have free ports". Two of three nodes hit it
during the Talos v1.12.11 upgrade; both needed hand intervention.
Selecting only `cluster` leaves `public` on the pod network, so mons, MDS,
MGR and the CSI pods keep Rook-managed identities and hold no host ports.
That removes the failure mode rather than working around it.
Isolation is not weakened. Rook's Multus prerequisites about hosts routing to
the Multus network apply to a Multus *public* network; selecting only
`cluster` means they do not apply here. VLAN 49 stays Ceph-internal traffic
and nothing else, with no router and no other participants.
MUST NOT be applied before phase A. Verified rather than assumed — kubectl
diff against the current cluster refuses it:
spec.network.provider: Invalid value: "multus": network provider must be
disabled (reverted to empty string) before a new provider is enabled
Second of the two rolling OSD restarts, bounded by the same two settings: one
OSD at a time, only once PGs are clean.
Follow-up, not in this commit: once this settles, the node addresses on VLAN
49 (10.99.0.1-.3) have no remaining purpose and should come out of the Talos
configs, after which the attachment's pool can take the whole /29 instead of
the three addresses it reserves today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Intermediate state, not a destination. san-network-attachment.yaml is already
in the cluster and stays unused until phase B sets provider: multus.
The detour is forced by Rook's own CRD validation, not chosen:
rule: self == '' || oldSelf == '' || self == oldSelf
message: network provider must be disabled (reverted to empty string)
before a new provider is enabled
host -> multus is rejected outright. It has to pass through empty, and an
empty provider means the default CNI — so addressRanges, which only takes
effect under provider: host, goes with it.
What this costs while it is in place: there is no cluster network. OSD
heartbeat, replication and recovery move onto the pod network and share it
with everything else instead of the dedicated X710 SFP+ link. Slower, and
every OSD restart pays that price in backfill. Not a correctness problem —
the OSDs can still reach one another throughout, which is the property that
matters, and the one a separate subnet would have broken.
The mons do not move here. Rook leaves existing mons on the network they were
created with; they change only on failover, which is deliberate and separate.
Bounded by the two settings that landed ahead of this: one OSD restarts at a
time, and only once PGs are clean again. Expect three sequential OSD restarts
with full recovery between them, and the same again in phase B.
kubectl diff against the live cluster shows provider host -> "" and the
generation bump. The removal of addressRanges does not appear there, because
this resource carries no managedFields and client-side diff cannot see a
field deletion. It is inert either way — Rook does not read addressRanges
without provider: host — but worth confirming after the apply rather than
assuming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rook's defaults are wrong for a cluster this size, and nothing here had
overridden them:
storage.osdMaxUpdatesInParallel default 20
upgradeOSDRequiresHealthyPGs default false
Twenty parallel OSD restarts is a small fraction of a large cluster. Here it
is all three, at once, with no gate on PG recovery in between. Every PG goes
inactive and every pool — RBD volumes and both CephFS mounts — becomes
unreachable for the duration. Not a degraded cluster, a stopped one.
With three OSDs at size 3 there is no spare copy to absorb the second
restart, so the health gate between one OSD returning and the next going down
is the only thing separating a rolling change from an outage.
This binds any change that rewrites the OSD deployment spec, not only Ceph
version upgrades. The immediate reason is the pending network provider
switch, which is exactly such a change — but the exposure predates it and
would have applied to any of them.
Lands inert: `kubectl diff` against the live cluster shows the two added
fields and the generation bump, nothing else. No OSD deployment spec changes,
so nothing restarts.
Deliberately separate from the provider switch rather than bundled with it,
so the bound is in place and observed before the operation it exists to
bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was 10.99.1.0/24, a separate subnet on the same segment, chosen to
avoid carving up a /29 with only three addresses left. That choice would have
broken the migration it exists for.
OSDs move one at a time, so the cluster runs mixed: an OSD already on the
attachment has to reach an OSD still using its node's host address on
10.99.0.x. There is no router on this segment — by design, it carries
Ceph-internal traffic and nothing else — so a pool outside the /29 is
unreachable from the un-migrated side. The two OSDs would mark each other
down, during precisely the window that is supposed to be the safe one.
Moves the pool into 10.99.0.0/29 with range_start/range_end reserving .4-.6,
leaving .1-.3 to the nodes. That is one address per OSD with nothing spare,
which is sufficient rather than merely tolerable: osdMaxUpdatesInParallel
bounds the migration to one OSD at a time, so allocations grow 1, 2, 3 and
never overlap. Headroom returns after the switch is confirmed and the node
addresses come out of the Talos configs — then the pool takes the whole /29.
Verified: the embedded CNI JSON parses with the intended range and bounds;
the NAD passes a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 of moving Ceph's cluster network off host networking. Inert on its
own: a NetworkAttachmentDefinition does nothing until a pod references it,
and nothing will until cephcluster.yaml names it under
network.selectors.cluster.
Why this exists at all: `addressRanges` only takes effect under
`provider: host`, so carrying replication over the dedicated 10G link meant
setting a provider that is not per-daemon. It puts mons on host networking
too, which makes a mon's address the node's DHCP lease — and when that lease
moves the mon cannot bind. Three mons on three nodes then each hold the mon
host ports, so Rook cannot schedule the replacement it just decided to create
and the failover stalls. Both happened during the v1.12.11 upgrade, on two of
three nodes.
None of that serves the goal. Per the Ceph documentation the cluster network
carries OSD heartbeat, replication and recovery only — "Ceph Monitors always
operate on the public network". Attaching it here gives the OSDs the link and
leaves every other daemon on the pod network.
Its own subnet rather than the existing /29. The nodes hold 10.99.0.1-.3 for
as long as Ceph still uses host networking, which leaves exactly three
addresses for three OSDs — no headroom during the one phase where things
actually move, and a leaked allocation from a hard node loss would mean an
OSD without an address. OSDs never need to reach the node addresses, since
the cluster network is OSD-to-OSD, so the pool sits in its own range on the
same segment instead. A /24 for three addresses costs nothing: nothing else
can ever live on this segment, and it takes address count off the table when
OSDs are added.
After the provider switch is confirmed, the node addresses on VLAN 49 have no
remaining purpose and can come out of the Talos configs.
No gateway and no routes, because there is no router on this segment and
there must not be one. That also means no `sbr` chaining, unlike the v30iot
attachments — there is no DHCP-supplied default route to confine here.
whereabouts for IPAM: dhcp needs a server that must not exist on the SAN, and
host-local keeps state per node with no cluster-wide coordination, so two
nodes would hand out the same address on shared L2.
MTU 9000 matches the link. macvlan inherits it, but stating it means a change
to the parent cannot silently halve replication throughput with nothing in
the tree recording the expectation.
Verified: whereabouts from step 1 is running on all three nodes with its
binary installed next to macvlan; the NAD passes a server-side dry run; the
embedded CNI JSON parses and carries the intended master, mtu, ipam type and
range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for moving Ceph's cluster network off host networking. Nothing
references whereabouts yet, so this lands inert: no existing attachment
changes, and the DaemonSet only installs a CNI binary that nothing invokes
until a NetworkAttachmentDefinition names it.
The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by
design. Neither IPAM already available covers it:
dhcp cni-dhcp-daemon works where a DHCP server is on the wire.
There is none on the SAN and there must not be one.
host-local installed by cni-plugins-installer, but it keeps allocation
state per node with no cluster-wide coordination. On a shared
L2 segment two nodes hand out the same address.
Whereabouts coordinates through the Kubernetes API — allocations live in the
ippools and overlappingrangeipreservations CRDs. Nothing about assigning an
address touches the segment being addressed, which is the property that makes
it usable on an isolated SAN.
The DaemonSet runs hostNetwork and privileged. That is upstream's shape, and
it matches what multus-daemonset.yaml, cni-dhcp-daemon.yaml and
cni-plugins-installer.yaml already do: the pod writes a CNI binary and its
config into /opt/cni/bin and /etc/cni/net.d, which is node plumbing, and it
has to work before the attachment it serves exists. It is the fourth instance
of an existing exception, not a new one.
The reconciler is split out deliberately. It needs no host access — only the
API, where the allocations are — so it runs as an ordinary pod-network
Deployment. It matters more than it looks: the normal release path is the CNI
DEL when a pod goes away, which does not run when a node is lost hard, and on
a /29 a handful of leaked addresses is the difference between an OSD getting
one and not.
Vendored from upstream v0.9.4 with the image tag pinned — upstream ships
`:latest`. Tag rather than digest, matching the other images in this
directory. The three CRDs are vendored unmodified so a future upstream diff
stays readable. nodeslicepools is included because the ClusterRole references
it, though the node-slice controller itself is not deployed.
Verified: kustomize builds; all nine resources pass a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodes run v1.12.6; v1.12.11 is the last patch of the 1.12 series. Upstream
tests migration only between adjacent minors and recommends going to the latest
patch of each intermediate minor, so 1.12.11 is the required stop before 1.13,
not an optional one.
Adds siderolabs/i915 while the image is being rebuilt anyway. Nothing requests
/dev/dri today — every manifest was checked, the only devices any workload asks
for are Ceph RBD and the NVMe disks by-id. It is included for the reason
thunderbolt already is: the extension set is fixed at install and upgrade time,
so adding it later would mean another rolling upgrade of all three nodes rather
than a config change. The MS-01's Alder Lake-P Iris Xe is present in hardware
either way.
New schematic 0f6fb8bd…, identical to a21cec8f… plus i915. Registered with the
Image Factory and the resulting installer image verified to exist for v1.12.11.
The absences are deliberate and now written down, so they are not re-derived as
gaps later: the X710 (i40e) and I226 (igc) NICs need no firmware extension —
intel-ice-firmware covers E810/ice only; mei is a prerequisite for Intel Arc
discrete GPUs and there are none; iscsi-tools, nvme-cli and util-linux-tools
have no consumer, since storage is Ceph RBD and device health comes from Ceph
rather than a node-level exporter.
Both runbooks move to the new schematic and version.
Note for whoever applies this: talosctl upgrade --image does NOT write
.machine.install.image, so this config has to be applied separately or the node
keeps pointing at whatever it was installed with. That is how the running nodes
ended up on ghcr.io/siderolabs/installer:v1.12.6 despite this file naming a
Factory image — which is also why talosctl get extensions is empty on all
three.
Kubernetes is untouched; talosctl upgrade does not move it. It stays on v1.35.2
and is a separate upgrade-k8s step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sockets bound the wildcard, so they accepted datagrams on eth0 as well as
net1. The namespace CNP grants every pod in home-assistant unrestricted access
to every other pod on every port, so any sibling pod could reach :5353 over the
cluster network and have records forwarded into HA's zeroconf cache. The
header comment described only the IoT-VLAN side of that.
so-bindtodevice=net1 confines delivery to datagrams that actually arrived on
the IoT VLAN interface. Binding to net1's address is still not possible — it is
DHCP-leased and unknown at exec time — but binding to the device gets the same
confinement without needing the address, and needs no capability: verified that
SO_BINDTODEVICE succeeds as uid 65534 with all capabilities dropped. Also
verified it costs no reception: 6 packets in 12s with the option, 6 in 12s
without.
Scope of what this fixes, stated exactly: the relay is not what makes HA's
zeroconf reachable from inside the cluster. HA binds :5353 on its pod address
(v4) and the wildcard (v6) — confirmed in /proc/net/udp and /proc/net/udp6 —
and the namespace CNP already permits any sibling pod to reach it directly.
This removes a second route to that listener, not the first.
Adds ipv6-v6only=1 on the v6 socket. The two containers share the pod's network
namespace, so a dual-stack v6 socket overlaps the v4 container's port. Measured
behaviour without it was no contention — both bound cleanly and the v4 receiver
kept taking packets throughout the v6 container's lifetime — so this is not a
latent bug being fixed, but the outcome depended on the kernel's bindv6only
default rather than on anything the manifest said. The roles are now explicit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The relay joined 224.0.0.251 only. mDNS is two protocols, and a socket joined
to one group hears nothing of the other, so every IPv6 announcement was
dropped — and a device that announces only over v6 would have stayed exactly
as invisible as before the relay existed.
Not hypothetical for this cluster: HA's zeroconf holds ff02::fb on eth0
alongside the v4 group, verified in /proc/net/igmp6. The listener was already
waiting for traffic the relay was never going to send it.
Adds a second socat container for the v6 group. Two processes rather than one
because socat carries a single address pair per invocation and the families
need different receive and send addresses. They fail independently and the
kubelet restarts each on its own.
Joining ff02::fb needs no address configuration on net1. The group is
link-local, so the kernel's automatic link-local address on the macvlan is
enough to receive on it; whether v30iot hands out global v6 addresses does not
affect reception.
The delivery Service becomes dual-stack (PreferDualStack with both families
listed, matching mosquitto-iot in this namespace), so each family is forwarded
over its own transport instead of folding v6 announcements into a v4 hop.
UDP4-SENDTO takes the A record and UDP6-SENDTO the AAAA from the same name.
Verified: the v6 socat address syntax parses and joins; a server-side dry run
allocates both families to the Service; all three manifests still apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HA's zeroconf listener joins 224.0.0.251 on eth0 and hears nothing. The HA
pod has received 0 multicast frames against 1.9M packets total. Two
independent reasons: multicast is link-local, so IoT-VLAN announcements never
reach the pod CIDR; and Cilium runs tunnel mode (VXLAN) with no
`enable-multicast`, so it does not replicate multicast between pods either.
The visible cost is that the ESPHome integration dials each device at the
address cached in its config entry and has no way to learn when that address
changes. A rotated lease produces `[Errno 113] Connect call failed` until
someone repoints the entry by hand. All eight ESPHome config entries were
created by `source=zeroconf` and now carry `discovery_keys: dhcp` — both
paths are inert today, so the addresses are frozen at whatever they were when
last discovered.
An avahi-style reflector cannot fix this. It receives on one interface and
re-emits as multicast on the other, and the second half is exactly what the
CNI drops. The published recipes for this all put avahi on the node, which
Talos does not permit.
So the relay bridges the last hop as unicast, which is the one direction the
CNI carries. It holds the multicast membership on v30iot through its own
macvlan and forwards each datagram to HA's :5353.
This works because python-zeroconf does not filter on how a packet arrived —
verified against _listener.py: datagram_received checks size, deduplicates,
and parses, with no test on destination address, source address, or source
port. Unicast mDNS is RFC 6762's QU mechanism, so this stays inside the
protocol.
Scope of the fix, stated plainly: passive reception only. RFC 6762 requires a
responder to re-announce when its address changes, which is the failure this
addresses, and new devices announcing themselves are seen too. Queries HA
sends still leave on eth0 and die there, so nothing that depends on
soliciting a response is restored.
No macvlan on the HA pod itself — that stays off-table, and the outcome since
it was removed supports keeping it that way. A separate relay was already the
recorded intent; this is that relay.
Shape follows shelly-manager-iot, which is the working precedent for L2
presence on v30iot: macvlan with DHCP IPAM, chained with sbr so the leased
default gateway cannot displace eth0's route. The forward path runs over eth0,
so that route has to survive.
Delivery target is a dedicated ClusterIP rather than the pod. socat resolves
its target once at start and cannot chase pod IPs; a ClusterIP is stable and
Cilium re-DNATs to the current backend. Kept separate from the
`home-assistant` Service, which fronts the nginx sidecar and is the XFF trust
anchor.
Exposure, since it is not zero: socat binds :5353 on all interfaces including
net1, because net1's address is DHCP-leased and unknown at exec time. Any host
on the IoT VLAN can send it a datagram that reaches HA's zeroconf cache — the
same trust already extended to real mDNS from that segment, which any device
there can equally forge. macvlan traffic bypasses Cilium policy, so the
namespace CNP does not constrain that interface. The container runs socat and
nothing else.
No CNP change needed: home-assistant-ns.yaml already allows intra-namespace
traffic on all ports.
Verified before committing: the socat address syntax joins the group and reads
in a loop without forking per datagram (UDP4-RECV, not UDP4-RECVFROM); it runs
as uid 65534 with no privileged operation; the image digest is pinned from the
registry; kustomize builds; all three manifests pass a server-side dry run.
Not yet verified, and only observable once deployed: that HA acts on the
forwarded records. The check is `discovery_keys` flipping back to zeroconf, or
a device surviving a lease change without hand-editing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotated turn_secret and relay_secret in production; both are correct and stay
rotated. proxy_token was in the same set and should never have been: rotating
it locked every proxy out of the management server.
The misjudgement was specific. proxy_token appears in proxy.env on the server
AND in proxy.env on each proxy, both Ansible templates, which looked exactly
like the symmetry test the other two pass. It is not: the management server
never reads it from a file Ansible writes. The token is registered in the
management DATABASE, so rotating it changed only what the proxies present.
management: proxy auth failed: rpc error: code = ResourceExhausted
desc = too many failed authentication attempts (~75/min, steady)
proxy: http: TLS handshake error ... unknown domain "home.renner.uno"
The second follows from the first — a proxy that cannot authenticate never
receives its domain configuration, so it rejects the domains it holds valid
certificates for.
Recovered by restoring the original proxy_token from the pre-rotation vault and
re-applying both roles. Auth failures went to zero and the domain errors
stopped. turn_secret and relay_secret keep their new values; both appear in
management.json.j2, so both ends really are Ansible's.
vault_netbird_proxy_token moves back into vault.yml. It was in
netbird-rotatable.yml, which this play rewrites wholesale — the next run would
have silently dropped it. Verified by re-running both deploys: changed=0.
Adds the test that would have caught this, in the play's own words: a secret is
safe to rotate here only if it appears in management.json.j2. Templating both
sides of a wire is not the same as owning both ends of a trust relationship.
Also corrects the shadow-key guard. vault.yml carried BOTH
vault_netbird_turn_secret and a bare netbird_turn_secret; the role default maps
one to the other, but a group var outranks a role default, so the bare key won
and rotating only the prefixed one would have changed nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Anything holding a long-lived TCP session to a device outside the cluster keeps
a half-open socket when the node under it moves. The peer drops its side; the
pod never learns, because no RST arrives and no keepalive fires.
This cost real time after the conversion. Home Assistant held seven ESTABLISHED
sockets to ESPHome devices that were themselves reporting "not connected". It
logged nothing and retried nothing, because as far as it knew the connections
were fine — so the search went through logs that were empty by construction.
The one device Home Assistant did log was the only one genuinely offline.
Records the check that settles it in one command, reading /proc/net/tcp in the
pod, and notes that reloading the integration is the lighter fix than
restarting the pod.
Placed in "Verify and restore" rather than the known-stale list: it is something
to check before calling a conversion finished, not a trap in the tooling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
common.yml applied monitoring-agent (Alloy) to every inventory host, pointing at
loki.srv.renner.uno:3100 and prometheus.srv.renner.uno:9090. Both moved into the
cluster and neither name resolves any more, so nas-1, claude-ws, netbird and the
proxies have been shipping logs and metrics nowhere.
Repointing them was the obvious fix and the wrong one. The cluster endpoints are
authenticated — loki.renner.uno returns 401 and prometheus.renner.uno sits behind
an Authentik forward-auth gate — while monitoring-agent has no auth support at
all. Making it work would mean per-host credentials, htpasswd entries in the
cluster secret, and some answer for remote-write past the gate. Nobody reads
those metrics: edge-1 is the host that matters, and edge-2 when it exists.
edge hosts are unaffected. install-edge-host.yml is self-contained and
deliberately never carried monitoring-agent; they ship via edge-log-shipper,
which already does this correctly — authenticated push to the cluster ingress
with a per-host credential.
install-netbird-monitoring.yml goes too; applying monitoring-agent to
netbird_server was its only content.
Comments referring to the role are corrected in install-edge-host.yml,
install-claude-workstation.yml, the claude-workstation role and its RUNBOOK.
Three more described the old srv Loki as "firewalled off the IoT VLAN" — it is
not firewalled, it is gone. The reason edge hosts use the authenticated ingress
is unchanged, but the stated reason was no longer true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/all/vault.yml held 35 keys. Nine are referenced by the surviving
tree, all NetBird:
netbird_oidc_client_id vault_netbird_auth_client_secret
netbird_oidc_client_secret vault_netbird_datastore_key
netbird_oidc_issuer vault_netbird_proxy_token
netbird_turn_secret vault_netbird_relay_secret
vault_netbird_turn_secret
The other 26 belonged to Proxmox, n8n, owncloud, the grafana and gitea OIDC
pairs, woodpecker, SNMP, cloudflared, the UniFi Poller account and both Home
Assistant tokens — services that now run in-cluster with their own credentials,
or no longer exist at all.
Edited YAML-aware rather than line-based: at least one value was a multi-line
block, which a line delete would have corrupted.
Verified the nine survivors are byte-identical, by comparing a normalised hash
of just those keys before and after, and again after re-encrypting:
e9fbbcef4127b84289ebf32e3b99e59a68e96e275af3c8c17d8ad0414a08e790
Same hash all three times. Decryption happened in a 0600 file outside the repo,
shredded afterwards; no plaintext touched the working tree.
Removing a key here does NOT revoke the credential. Anything still live in its
target system needs revoking there — that is a separate exercise, and the Home
Assistant tokens in particular are unreferenced here but valid in HA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Missed in the Proxmox removal. I built that list from a `hosts:` listing I had
truncated with `head -40`, so anything past the fortieth line was never
considered — a self-inflicted blind spot rather than a judgement call.
install-postgres-monitoring.yml authentik-db-1.srv.renner.uno and
prometheus.srv.renner.uno, both NXDOMAIN
configure-audiobookshelf.yml audiobookshelf.srv.renner.uno is NXDOMAIN;
audiobookshelf runs in-cluster
The postgres-exporter role goes with them — install-postgres-monitoring.yml was
its only consumer.
Found by cross-referencing every vault key against the surviving tree rather
than by looking at plays again: vault_postgres_exporter_password came back as
"still referenced", which was only true because a dead play still referenced it.
That check now leaves the group_vars/all vault holding nine keys anything uses,
all NetBird, and twenty-six nothing does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The play generated secrets with community.general.random_string. The CI image
(ci/images/ansible/Containerfile) installs only community.proxmox and
community.docker, so the play failed at its first task when run through the
project's own tooling. It resolved on a workstation that happened to have the
collection, which is exactly why verifying it there proved nothing.
Replaced with ansible.builtin.password, which needs no collection and produces
the same thing: 48 alphanumerics, generated without persisting (path /dev/null).
Also hardens the shadow-key guard. It piped `ansible-vault view` into `grep -c`
with failed_when: false, so a failed decrypt — wrong password, moved file —
sent empty input to grep, which reported zero matches. Indistinguishable from a
clean vault, meaning the guard would have waved the rotation through precisely
when it knew least. A preceding task now proves the vault is readable, and the
assertion requires both that proof and the zero count.
Corrects the openbao rotation path in the header comment; it is under
kubernetes/.
Caught by the review-bot on #626.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.woodpecker/update-all.yml ran `ansible-playbook -i inventory/ plays/update.yml`
on a cron. This branch deletes that playbook, because every playbook it
imported targeted Proxmox hosts or guests that no longer exist, so the next
scheduled run would have failed outright.
Removing the pipeline rather than repointing it: there is nothing left for it
to update. Its imports were update-vm, update-lxc, update-authentik,
install-n8n, install-grist, install-music-assistant, update-docker and
update-pve-rolling — all removed here, all targeting hosts that no longer
resolve.
Swept every pipeline for references to deleted plays. The five still named
(install-claude-workstation, install-garage-synology, install-netbird-client,
install-netbird-proxy, install-netbird-server) all survive.
Also from the review:
common.yml `all:!pbs` was a no-op I introduced when simplifying the
old exclusion list — no pbs group exists. Now plain
`all`.
qemu-guest-agent dropped the `pve_cluster`/`proxmox_nodes`/
`proxmox_all_lxc` group guards; those groups are gone,
so the conditions were always true.
Caught by the review-bot on #625.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/<group>/ files merge alphabetically and the last one wins.
netbird-rotatable.yml sorts before vault.yml, which still defines
vault_netbird_turn_secret, vault_netbird_relay_secret and
vault_netbird_proxy_token — verified, all three are present.
The rotation itself would have worked, since the plays pass the new values
explicitly and play vars outrank group vars. The damage came later: the next
ordinary run of install-netbird-server.yml or install-netbird-proxy.yml would
read the stale value from vault.yml and push it back out, breaking
server<->proxy auth. Those deploys run unattended on changes to the roles, so
it would have surfaced on something unrelated to secrets.
Adds a pre-flight check that counts — never prints — matching keys in vault.yml
and refuses to rotate while any remain, plus the one-time migration step in the
header.
Caught by the review-bot on #626.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotates the three shared secrets where Ansible controls both ends, so the two
sides cannot disagree:
vault_netbird_turn_secret turnserver.conf (coturn) + management.json, one
role, one host
vault_netbird_relay_secret management.json on the server + relay.env on the
proxies
vault_netbird_proxy_token proxy.env on both
Opt-in, because rotation is not idempotent and briefly interrupts the mesh:
ansible-playbook plays/rotate-netbird-secrets.yml -e rotate_confirm=yes
New values are written to a dedicated encrypted file rather than edited into
the main vault. The plaintext is rendered at 0600 with a .new suffix, encrypted,
and only then moved into place, so a failure at any step cannot leave readable
secrets where the inventory would load them. The play asserts the installed
file carries the $ANSIBLE_VAULT header before proceeding.
Two are deliberately excluded and the play says why:
vault_netbird_datastore_key encrypts the datastore at rest; changing it
without dump/restore makes existing data
unreadable
vault_netbird_auth_client_secret its other half is in Authentik, so rotating
only the consumer breaks login
These are Ansible's own credentials, not copies of the cluster's. OpenBao holds
separate credentials for some of the same systems, deliberately different values
— one consumer, one credential, independently revocable. This play does not
touch OpenBao and is not meant to.
Proxies roll one at a time so a relay stays reachable. The server is
reconfigured first, so there is a short window where a not-yet-updated proxy
fails to authenticate; clients reconnect and peer-to-peer tunnels are not torn
down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing here had a target left. Every removal is backed by a probe rather than
an assumption:
pve_cluster (jupiter/neptune/saturn) run Talos; no SSH at all
authentik-db-1/2, authentik-app-1/2 NXDOMAIN
grafana.srv, vault.srv, paperless.srv gone (grafana/paperless resolve but
answer neither ping nor SSH)
n8n.srv, grist.srv, wiki.srv NXDOMAIN
owncloud.srv, ical-proxy.srv, loki.srv NXDOMAIN
proxmox_* dynamic groups api.proxmox.yaml queried
jupiter.renner.uno:8006
24 plays, 14 roles and the Proxmox dynamic inventory go. update.yml goes with
them: every playbook it imported is in that list, so the aggregate updater had
no fleet left to update.
Kept, because their hosts answer: netbird server and proxies, nas-1, edge-1 and
its capability groups, claude-ws. common.yml and install-node-exporter.yml lose
exclusion clauses naming groups that no longer exist.
NTP IS NO LONGER HIGHLY AVAILABLE, and this change records that rather than
causing it. ntp_cluster was jupiter (MASTER, prio 100), saturn (90), neptune
(80) and edge-1 (BACKUP, 70); the conversion took the first three. A single
VRRP member holds the address regardless of configured state, so fd40::123 is
still served — verified answering — but by one thin client with nothing behind
it. install-ntp-ha.yml still carries the four-host priority map, now describing
hosts that do not exist. Restoring redundancy needs a second chrony host that
is not a Talos node.
The gatus role is removed entirely, which supersedes the narrower edit in the
DNS cleanup that dropped its Proxmox DNS check.
Left alone: plays/archived/ still references pve_cluster, which is what an
archive is for. paperless-migration stays because an archived play uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The removal has been applied:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed, 1 forgotten.
`tofu state list` now returns no Proxmox resources at all, and a full plan with
refresh enabled produces no Proxmox errors — the "no route to host" against
10.2.40.201:8006 that failed every infrastructure plan since the conversion is
gone.
The block was always one-shot. Left in place it refers to a state entry that no
longer exists, which is harmless but misleading: it reads as pending work.
Worth recording for the next time this shape comes up, because it cost a failed
merge to learn. A `removed` block alone does not fix an unreachable provider:
OpenTofu still refreshes the resource before it can forget it, and that refresh
is exactly what cannot succeed. It takes one run with `-refresh=false` to
process. `-target` does not help — targeting does not reach `removed` blocks, so
a targeted plan reports "No changes" while the state entry sits there untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Proxmox DNS HA VIP (10.2.40.53) was decommissioned after it dead-ended
non-Ingress resolution and broke ~1374 Home Assistant entities on a pod restart
(2026-07-14). With Proxmox itself now gone, the implementation and everything
pointing at it are dead weight — and one piece was actively dangerous.
Removed:
dns-server/ the Go implementation, its systemd unit,
and a 7.4 MB checked-in binary
opentofu/scripts/remove_unifi_clients.sh
the "Proxmox DNS (VIP)" Gatus check in the ansible role
remove_unifi_clients.sh is the one worth calling out. It instructs the reader
to deploy module.proxmox_dns and point UniFi DNS at 10.2.40.53 — precisely the
configuration that caused the outage. It also works by `tofu state rm`, which
orphans resources rather than removing them. It was already marked deprecated
in its own header; anyone who ignored that and followed it would have
reintroduced a known failure.
The Gatus check queried k8s-1.srv.renner.uno against 10.2.40.53 every 30s. It
belongs to the ansible gatus role, which targets a Proxmox LXC that no longer
exists, so nothing was running it — but the check itself asserts the decommis-
sioned server should answer, which is the opposite of the intended end state.
Deliberately kept: the three comments in bootstrap.sh, coredns-config and
k8s-gateway that record 10.2.40.53 as DECOMMISSIONED and say not to reintroduce
it. They carry the outage date and blast radius, and are the reason this keeps
not happening again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every infrastructure plan has failed since neptune was wiped. module.k8s_3 still
declared a Proxmox VM, so the bpg/proxmox provider tried to reach a hypervisor
that no longer exists:
Error: failed to get resources list of type ("vm") for cluster:
dial tcp 10.2.40.201:8006: connect: no route to host
with module.k8s_3.proxmox_virtual_environment_vm.this
Deleting vm_152_k8s_3.tf alone does not fix that. Removing a resource from
configuration makes OpenTofu plan a destroy, and planning a destroy refreshes
the resource through the same unreachable provider.
A `removed` block with `lifecycle { destroy = false }` forgets the state entry
without contacting Proxmox. Nothing is orphaned: the guest, its disks and the
hypervisor underneath were all destroyed during the conversion. This is the
declarative equivalent of a state removal, reviewable in git rather than run by
hand against production state.
talos_image.tf goes with it. Its locals named the nocloud image for VM import
and had no consumer other than the VM module.
Deliberately kept for a follow-up: the proxmox provider block, the
proxmox_vm/proxmox_lxc/bpg_proxmox_container modules and sync_state_from_pve.py.
A provider with no resources is never contacted, so leaving it costs nothing,
and removing it should wait until a clean plan proves state holds no other
Proxmox resources. That check cannot be made from here — the S3 credentials
used for local runs have been revoked.
Verified with `tofu validate` and `tofu fmt -check` offline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
machine.install.image was never set, so every node installed from the stock
ghcr.io/siderolabs/installer that talosctl gen config defaults to. Extensions
live in the installer image, which makes the schematic a property of the boot
media alone — nothing lands on disk. talosctl get extensions returns empty on
all three nodes.
Two extensions are missing as a result, and the cold-rebuild runbook explains
why each was chosen for bare metal:
siderolabs/intel-ucode CPU microcode. On a VM the hypervisor supplied it;
on bare metal nothing does except the BIOS.
siderolabs/thunderbolt Inert until machine.kernel.modules asks for it.
Present so enabling the MS-01 mesh later needs
neither a new image nor a reinstall — precisely the
property installing without it destroys.
Set in the shared cluster patch rather than per node: all three are bare metal
now and want the same image. Schematic a21cec8f is the bare-metal one from
RUNBOOK-cold-rebuild.md — same net.ifnames=1 as the Proxmox schematic, minus
qemu-guest-agent.
Config-only. install.image is read at install and upgrade time, so no running
node changes until it is upgraded; the three nodes stay without extensions
until a deliberate rolling upgrade is scheduled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cilium reads datapath options once, at agent start, and the chart puts no
config checksum on the pod template. A changed ConfigMap therefore leaves every
agent running the previous datapath until something restarts it, and nothing
does — the DaemonSet is not rolled by a ConfigMap-only Helm change.
This already cost an outage. vlan-bpf-bypass landed in cilium-config and every
v30iot macvlan workload stayed broken until the agents were deleted by hand,
one node at a time (issue #592). The merge looked successful: HelmRelease
Ready, release upgraded, key present in the ConfigMap, and nothing working.
policyMapMax carries the same exposure. Its comment says "takes effect on agent
restart", which is documentation standing in for automation — as does the
equivalent note in the two bootstrap copies.
Reloader is already deployed and used across the repo for the same purpose.
Named per-ConfigMap rather than blanket, matching the existing convention.
The trade, stated plainly: any cilium-config change now rolls the CNI
cluster-wide, node by node. That is the same disruption a datapath change
already required; the difference is that it happens on its own rather than
being forgotten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step pushed a nocloud Talos image onto each Proxmox node so the k8s VMs
could boot from it. No Proxmox node remains — neptune was the last one, and all
three control-plane nodes now boot the metal image from their own installed
disk.
Left in place the step would call the Proxmox storage API for a host that no
longer answers it, then sit through its 60x10s download-wait before giving up,
on every infrastructure apply.
The block's own comment already scheduled its removal: "the whole step goes away
once neptune is converted too."
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every v30iot macvlan workload on a bare-metal node has been stuck in sandbox
creation since its node was converted. Cilium's datapath drops VLAN-tagged
frames unless bpf.vlanBypass lists the tag, and cilium-config carried no
vlan-bpf-bypass key at all.
Inside the Proxmox guests ens19 was an untagged virtio NIC on the v30iot
bridge, so nothing was tagged by the time Cilium saw it. On bare metal ens19 is
a VLAN 30 sub-interface of enp2s0f0np0, and the macvlan children's DHCP
exchange never reaches the wire.
Measured rather than inferred: cilium_drop_count_total on the affected node
reports ~123k INGRESS and ~1.1k EGRESS drops with reason "VLAN traffic
disallowed by VLAN filter", against none on the node that was still a VM, and
node_network_receive_packets_total for ens19 reads 0 there against 10 pkt/s on
the VM. Full analysis in issue #592.
The symptom hides its own cause. A tc-layer drop surfaces to an AF_PACKET
sender as ENOBUFS, so the CNI dhcp plugin reports "no buffer space available"
and "network is down" instead of a DHCP timeout.
VLAN 40 is deliberately not listed. mgmt0 rides the same parent and carries the
node IP, etcd and the API, and it passes today without a bypass entry, so
adding it would change a working control-plane datapath to fix an IoT one.
All three copies of the key move together — the HelmRelease, kubernetes/talos/
cilium.yaml, and the inline bootstrap manifest in patches/cilium-cni.yaml — so
a cold rebuild does not come up without it.
The runbook predicted this exact failure and is updated to record that it
happened, along with how to recognise it from the drop counters rather than the
misleading CNI error. music-assistant's comment claiming ens19 is "the v30iot
NIC" on every node is corrected; it is a NIC on the edge worker and a VLAN
sub-interface on the control-plane nodes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.
The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.
The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.
cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.
cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three nodes are converted and the cluster is back to three OSDs across
three zones with 217 PGs active+clean, so the conversion window that justified
2/1 on this pool is over.
The min_size override goes with it. It existed only because Ceph does not
lower min_size when size is reduced, so the pool would have been 2/2 while a
host was out. At size 3 the default min_size of 2 is what the pool wants.
The comment loses its TEMPORARY banner and now states the standing rationale
and the condition under which the pool degrades to two copies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune is converted and back in the cluster, so k8s-3 returns to the storage
fleet on its new PM983. All three nodes now name their OSD by /dev/disk/by-id,
and the storage fleet is whole again.
The comment generalises to all three nodes and records neptune's serials.
k8s-3 is the clearest evidence for by-id addressing: it enumerated the Kingston
as nvme0n1 on one boot and the Crucial as nvme0n1 on the very next one, so a
kernel-name reference had no stable meaning on that machine at all.
The old osd.1 lived on the Crucial that is now EPHEMERAL. That drive was wiped
during the conversion, so the OSD is unrecoverable and is purged from the CRUSH
map separately; this change only creates its replacement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune now runs Talos directly, so k8s-3 moves from the VM-era layout to the
multi-document form k8s-1 and k8s-2 already use. The old config selected
interfaces by busPath against virtio NICs and installed to /dev/vda; neither
exists on bare metal.
Disks are selected by serial rather than kernel name. That is not defensive
styling: Talos enumerated this machine's drives differently than Proxmox did,
and differently again between two consecutive boots — nvme0n1 was the Kingston
on one boot and the Crucial on the next. A device-name reference would have
installed onto the wrong drive.
nvme 50026B7382B48B90 Kingston system disk
nvme 2326E6E8E67D Crucial EPHEMERAL, 83% of rated endurance
nvme S437NF0M501176 PM983 OSD, 3% used
Serials, interface names and drive health were read from the machine itself —
the Kingston and Crucial off the running Proxmox host before the wipe, and
confirmed again from Talos maintenance mode.
k8s-2.yaml loses the sentence claiming k8s-3 is still a Proxmox VM. No Proxmox
remains in the homelab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune is the last node to convert to bare metal, and its OSD drive is swapped
in the same window. Removing k8s-3 from the explicit node list first means Rook
stops managing that OSD before the drive it lives on goes away, rather than
recreating it mid-reconcile and backfilling onto a disk that is about to be
pulled.
The device comment for k8s-3 goes with it: it described the PCI-passed-through
drive that node saw as a Proxmox guest, which stops being true here.
Ceph runs on two OSDs and size-3 pools stay degraded until the node rejoins.
cephfs-metadata is already at 2/1 for exactly this window.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saturn is back on bare metal with the PM983 in place, so k8s-2 returns to the
storage fleet. Like k8s-1 it names the device by /dev/disk/by-id: the machine
has three NVMe drives and Talos enumerates them differently than Proxmox did,
so nvme0n1 there is the Kingston system disk.
The runbook prescribed applying the full per-node config in maintenance mode,
which hangs the install. A document referencing an interface that does not
exist yet leaves the node at STAGE: Installing with KUBELET n/a, the console
repeating
k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
which reads as a Kubernetes fault but is an install that never completed. On
saturn it was the ens19 VLAN document and the sysctl naming it. Three cold
installs hung; the identical config applied cleanly to the running node.
Wiping disks changes nothing.
Install now runs in two stages: a minimal network config plus EPHEMERAL, which
is one-shot and cannot be deferred, then the full config applied to the running
node behind --mode=try so a bad network config reverts itself.
Also corrects the claim that maintenance mode cannot be inspected. talosctl get
works there; --insecure is a subcommand flag and has to follow `get`, otherwise
cobra resolves the legacy `talosctl disks` command and rejects the flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VM 605 is pinned to saturn, so it cannot survive the conversion, and it has to
go before the node leaves the Proxmox cluster — afterwards the provider cannot
reach it and every plan errors.
Nothing uses it. Its purpose was cloning new Proxmox VMs, Proxmox is being
retired, and the one remaining VM definition sets template_name = null
explicitly rather than cloning from it.
The module default is changed to null for the same reason: it named a template
that no longer exists, which would only surface as an apply-time failure for
whoever created the next VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saturn is being converted to bare metal, so its VM definition goes and it
leaves the list of Proxmox nodes the pipeline seeds the nocloud image onto.
The apply must run while saturn is still a Proxmox cluster member. Once it
leaves, the provider cannot reach the node, the plan errors, and the only way
out is tofu state rm.
module.k8s_2 is referenced nowhere else in opentofu/, so this destroys exactly
one resource.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Structurally identical to k8s-1.yaml; only serials, MACs and addresses differ.
Written from saturn's live hardware rather than assumed: the X710 port 1 MAC
came from the guest that currently owns it, since the host cannot see a
passed-through NIC.
Kingston 50026B7382DA5F6E system disk
Crucial 2332E86B74BC EPHEMERAL, retiring from osd.0 at 81% wear
X710 p0 58:47:ca:76:6f:73 trunk, VLAN 40 + VLAN 30
X710 p1 58:47:ca:76:6f:74 Ceph cluster network, 10.99.0.2/29
Links are addressed by kernel name rather than LinkAliasConfig, and both disks
by serial. Talos and Proxmox do not agree on NVMe enumeration order, and a
network document that fails to resolve takes the whole machine's DHCP with it.
Not applied by anything on merge — talos patches are applied by hand. Landing
it before the conversion so the file under review is the file that gets
installed.
Verified with talosctl validate --mode metal against v1.12.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the node from storage.nodes before purging its OSD is what makes the
purge stick. Under useAllNodes the operator rebuilt a purged OSD within one
reconcile and began backfilling onto the drive being replaced; listing nodes
explicitly exists so a node can be taken out first.
Re-add after the conversion naming the device by /dev/disk/by-id, as k8s-1
does. A bare device name is not safe on bare metal: Talos enumerates the three
NVMe drives in a different order than Proxmox, and nvme0n1 is the system disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pvecm expected 1 is the official recovery, documented verbatim in the storage
replication chapter of the Proxmox admin guide. Its caveat is about restoring
the missing node instead; here the missing node is deliberately gone for good,
so enforcing quorum is the correct branch rather than a workaround.
Also records that it must be run only once the node is alone -- corosync
rejects a value below the members it can see, which is why the attempt on the
current two-node cluster returned CS_ERR_INVALID_PARAM -- and that it is
runtime only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter is converted and its OSD rebuilt, so the capacity constraint that took
immich down is gone with all three nodes back.
cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be
degraded again for saturn and then neptune, and each transition re-replicates
the pool, so toggling it per node buys nothing but churn. Its comment said to
restore it once k8s-1 was back, which is now misleading about the plan; it
restores after the last conversion instead.
The immich comments lose their jupiter-specific framing — taking both
Deployments down is a pre-flight step for any single-node maintenance now, and
the runbook carries it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written from doing jupiter, with saturn and neptune still to go. The conversion
itself was straightforward; every failure came from state that outlived the VM,
so the runbook is weighted towards those rather than towards the happy path.
The three that cost real time:
A rebuilt node gets a new MAC and a new lease, and Ceph keeps its mon pinned to
the old address in the monmap, the mon-endpoints ConfigMap and the deployment's
--public-addr. The mon then cannot bind, the operator's next step wants to
restart a healthy mon, Ceph refuses because that would drop quorum, and it
retries every minute forever without reaching the storage reconcile. Fixing the
ConfigMap before the monmap and the deployment matters, or the operator
recreates the mon from the stale entry.
Talos stops running default DHCP for the whole machine as soon as one new-style
network document exists, and a LinkAliasConfig selector that matches nothing is
skipped silently. Together those turn a small config mistake into a node with no
address on any interface.
EPHEMERAL placement is honoured only on first provision, so it has to be right
in the first apply-config; afterwards it is ignored and relocating needs a wipe.
Also records the Proxmox quorum step that starts mattering at the second
conversion, the Secure Boot blocker, and that Talos and Proxmox enumerate the
same three NVMe drives in different orders — which is why the OSD is named by
by-id path rather than device name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter is back as a bare-metal Talos node, so its OSD returns to the storage
fleet on the new drive.
The device is named by /dev/disk/by-id rather than by bare device name, because
the bare name would be wrong here. Talos enumerates this machine's three NVMe
drives in a different order than Proxmox did, and nvme0n1 on k8s-1 is the
Kingston system disk:
nvme0n1 50026B7382B48B86 Kingston Talos system disk
nvme1n1 2332E86B749A Crucial Talos EPHEMERAL (/var)
nvme2n1 S437NX0M400426 PM983 this OSD
Rook would most likely have declined the system disk for carrying partitions,
but relying on that is luck rather than intent. The by-id path derives from
model and serial and does not depend on enumeration order.
k8s-2 and k8s-3 keep bare names: each guest sees exactly one block device,
the drive passed through from its host, so there is nothing to disambiguate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first install of this node came up with no address on any interface and had
to be recovered from the console. This is the configuration that replaced it,
now running on jupiter.
The alias layer was the cause, but not for the reason it first appeared. Reading
the v1.12.6 source: aliases ARE supported as VLAN parents — link_config.go
resolves parent links through the alias map, and the field's own doc comment
says so. The structure was fine. What is not fine is the failure behaviour.
LinkAliasConfigController skips a selector that matches zero links with no log
of any kind, contradicting the doc comment on the type which claims an error is
returned. And permanent_addr comes from the ETHTOOL_GPERMADDR ioctl with no
fallback to the current address, so a driver that does not answer it yields an
empty string, an unmatched glob, and silence.
That alone would have been survivable. What made it total is that Talos stops
running default DHCP operators for the WHOLE machine as soon as a single
new-style link, VLAN or DHCP document exists — the gate is presence of a
document, not success of resolution. So the silent miss removed the only
configured path while the safety net was already disabled. There is no fail-open
option.
Aliases are worth having and should come back. The condition is that they are
verified rather than assumed: apply with `--mode=try`, which rolls back on its
own, and check that linkaliasspec exists and that permanentAddr is actually
populated before trusting it. A LinkConfig on the real kernel name alongside the
alias resolves regardless of whether the alias landed, and costs nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three NVMe drives are present, not two: the Crucial stays in the machine as
EPHEMERAL rather than being pulled.
The second I226 is enp90s0, not enp89s0. Adding that third drive renumbered
the PCI buses and renamed it, which is a concrete demonstration of why every
NIC in this file is matched by permanent MAC rather than by name or bus path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Crucial P3 that was this node's Ceph OSD stays in the machine rather than
being pulled, and takes over /var. At 87% of rated endurance it is finished as
an OSD, but everything EPHEMERAL holds — containerd images and snapshots, pod
logs, the Ceph mon store — is reconstructible, so the drive is consumable in
that role.
It also removes the constraint that keeps biting elsewhere: /var becomes
~931 GB instead of the 64 GB the VM-era zvol gave it. k8s-3 tripped
DiskPressure twice today the moment it absorbed another node's pods, with
image GC already at its floor and ~20 GB of unreclaimable overlayfs snapshots.
This node will not have that problem, and the same treatment is available to
saturn and neptune when they convert.
Talos only honours volume configuration when the volume does not yet exist, so
this must ship in the first apply-config; applied later it is silently ignored
and relocating EPHEMERAL then needs a disk wipe and a boot from install media.
The selector matches an exact serial. There are three NVMe drives in this
machine now and `disk.transport == 'nvme' && !system_disk` would match the
PM983 as well, which would hand Talos the OSD drive.
Verified with talosctl validate --mode metal against v1.12.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With one of three nodes out for the bare-metal conversion, the remaining pair
cannot hold the workload: cluster CPU requests exceed two nodes' capacity by
roughly 200m once DaemonSets and unplaceable Ceph pods are discounted, and
k8s-3 sits ~5.8 GiB above the kubelet's eviction threshold with image GC
already at its floor.
immich is the right thing to stop rather than squeeze. machine-learning
reserves 500m against a 2m idle draw, but inference bursts hard, so cutting
its request would misrepresent what it needs the moment someone uploads
photos. Stopping it removes the reservation honestly, and takes its pods and
image pulls off k8s-3 for the duration.
The Postgres cluster stays up, so recovery is these two replica counts and
nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
useAllNodes makes it impossible to retire an OSD ahead of pulling its drive.
Purging osd.2 and deleting its deployment held for one reconcile: the operator
saw a node it was told to use, carrying a device with a BlueStore signature,
and rebuilt the OSD. Ceph then began backfilling onto the very drive being
replaced.
Listing nodes explicitly makes membership of the storage fleet a property of
this file rather than of which nodes happen to exist, so a node can be removed
before its disk is touched.
k8s-1 is out while jupiter's OSD is swapped for the PM983 and the node is
converted to bare metal. The cluster runs on two OSDs until it is re-added;
all pools are size 2 / min_size 1 and stay fully redundant across the pair.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Draining k8s-1 for the jupiter conversion left three pods unschedulable:
k8s-2 and k8s-3 both sat at 100% of requested CPU while actually using 21-23%.
Cluster-wide the gap is 19,680m requested against 7,957m used.
crowdsec is the clearest single contributor. The chart defaults reserve 500m
for each agent and for the LAPI; measured draw is 1-2m. As a DaemonSet the
agent alone parks half a core on every node. Nothing here had ever set
resources, so this was an inherited default rather than a decision.
Only the request changes. Limits stay at the chart default, so a log flood can
still burst to 500m; what shrinks is the reservation the scheduler subtracts
from every node whether or not the process uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the
MS-01's only U.2-capable slot means moving the boot drive anyway. Doing the
bare-metal conversion in the same window avoids opening the machine twice and
throws away no work: the osd-ssd PCI mapping that a Proxmox-side swap would
need to be re-pointed at ceases to exist.
Networking uses the v1.12 multi-document config rather than the deprecated
machine.network.interfaces. The docs do not define what happens when one link
is configured by both schemas, so k8s-1 keeps every link in one of them. This
also buys VLANConfig.name, which is free-form: naming the VLAN 30 link ens19
keeps the four v30iot macvlan NADs working unchanged across VM and bare-metal
nodes, so the cluster can stay mixed without pinning those workloads.
Physical NICs are matched by permanent MAC, not driver — both X710 ports bind
i40e here, unlike the single passed-through port k8s-2/k8s-3 see.
cephfs-metadata drops to size 2 / min_size 1 for the duration. At size 3 on
two remaining hosts every PG sits undersized with min_size 2, so one further
OSD loss would block IO on every CephFS mount cluster-wide. This must go back
to size 3 once k8s-1 rejoins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sweep restores every discovered Velero schedule, immich's ~180 GiB
library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and
measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per
week — ~47% of the endurance budget, against a ~65 GB/day weekday
baseline.
That baseline is itself the result of the 2026-08-18..21 write-reduction
work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs,
paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive
writes ~70% from ~195 GB/day. With the application tier down, the restore
test is now the single largest consumer of remaining QLC endurance; the
three OSD drives are at 81/82/87% used.
Monthly still exercises every restore path and cuts the sweep's share of
drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB.
Cadence-only change:
- 0 8 * * 0 -> 0 8 1 * *, keeping the 08:00 UTC slot that clears the
03:15-07:15 daily backup window (see the comment in the manifest).
- ttlSecondsAfterFinished 14 d -> 40 d, so the previous run is still
inspectable when the next one fires.
- Comments and docs that state the cadence updated to match, including
the BLUESTORE_SLOW_OP_ALERT runbook text, which tells the reader to
check whether a stall fell inside the restore-test window.
No day-of-week form is available: Kubernetes parses cron with
robfig/cron's standard parser, whose day-of-week field allows only
`* / , - ?` (no Quartz `#` nth-weekday), and restricting both day fields
ORs them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Wetterstation now carries a non-default reporting configuration: min
interval 5s on all ten reportable attributes, and tighter reportable-change
deadbands on temperature, humidity and pressure. None of it is expressible
in configuration.yaml - Zigbee2MQTT has no settings-schema field for
reporting configuration - so it lives only on the device and in Z2M's
database.db. This documents it and how to restore it.
min was 10s against an 8.8s transmit interval. Since min is a hold-off, every
change-triggered report was held past a transmit cycle and landed on the
next, so attributes reported at best every 17.6s and drifted out of phase.
The published stream showed no 8.8s periodicity at all. At min 5 each
attribute reports on its own transmit and the stream carries one publish per
cycle, measured at a mean gap of 8.69s. Publish volume fell from ~631/h to
~445/h because each publish is now a whole cycle rather than a fragment.
The deadbands come from the shared modernExtend helpers, which the WS90
definition composes without overrides. They are anti-chatter values sized for
battery-powered indoor sensors. Temperature was 1.00 C, giving a staircase
that moved about twice an hour; it is now 0.1 C, matching the resolution the
device actually reports. Pressure was 5 kPa - 50 hPa, which real barometric
pressure never traverses, so it only ever updated on the hourly heartbeat
that pressure_trend and weather_condition derive from. Note the same library
configures the identical ZCL attribute with a 5 hPa deadband in its older
reporting.ts helper, so the two code paths disagree by 10x.
Anything that makes Z2M re-run configure on the device reverts all of it, and
a fresh PVC will not have it. The restore commands are written out literally
rather than generated by a shell helper, because the nested quoting through
`kubectl exec -- sh -c` is where this goes wrong; each was executed and
verified as written, including the read-back, and re-applying is a no-op.
No behaviour change from this commit itself - the device was reconfigured at
runtime, this only records it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot
issue-fix-bot