fix(rook-ceph): ship per-driver CSI ServiceAccounts for v1.20 operator #12

2026-06-08T18:55:45Z

k.renner commented

2026-06-08 18:55:45 +00:00

Summary

The rook-ceph HelmRelease's floating >=1.16.0 constraint silently upgraded to v1.20.0 about 5 days ago, which extracts CSI driver management into the upstream ceph-csi-operator subchart. That subchart's generated ctrlplugin Deployments and nodeplugin DaemonSets reference per-driver ServiceAccounts (rbd-ctrlplugin-sa, cephfs-ctrlplugin-sa, rbd-nodeplugin-sa, cephfs-nodeplugin-sa) that the main rook-ceph chart no longer ships — they moved to a separate ceph-csi-drivers chart.

With the SAs missing, both ctrlplugin Deployments sat at 0/2 with FailedCreate for 5 days, blocking PVC attach/detach across the cluster:

monitoring/kube-prometheus-stack-grafana — stuck 2d21h on FailedAttachVolume
monitoring/alertmanager-...-0 — same, blocking the kube-prometheus-stack HR (Helm rollback also failed)
audiobookshelf/libation-... — same (cron pod, stuck 4d14h)
home-assistant/home-assistant-... — pinned to k8s-edge, which never got an RBD nodeplugin pod (DaemonSet couldn't create new pods due to missing SA)

monitoring-rules KS is dependsOn blocked by monitoring.

Why static manifests instead of the upstream chart

The ceph-csi-operator/ceph-csi-drivers chart is the documented Rook v1.20 fix, but it cannot be installed cleanly here: live helm-controller is v1.5.4, which predates disableTakeOwnership. Helm would refuse to adopt the operator-managed Driver and OperatorConfig CRs created by the v1.18 → v1.20 auto-migration, and the install would fail.

Instead, we render only the SAs + RBAC the operator needs from the chart's templates (v1.0.1) as static manifests under kubernetes/infrastructure/ceph-csi-drivers/. The existing operator-managed CRs remain the source of truth.

Changes

File	Change
`kubernetes/infrastructure/ceph-csi-drivers/serviceaccounts.yaml`	New — 4 SAs
`kubernetes/infrastructure/ceph-csi-drivers/rbac.yaml`	New — 16 RBAC resources (CR/CRB/R/RB per SA), rendered from ceph-csi-operator/ceph-csi-drivers chart v1.0.1
`kubernetes/infrastructure/ceph-csi-drivers/kustomization.yaml`	New
`kubernetes/infrastructure/kustomization.yaml`	Add `ceph-csi-drivers/` to umbrella
`kubernetes/infrastructure/rook-ceph/helmrelease-operator.yaml`	Pin chart `>=1.16.0` → `~1.20.0` to prevent the same floating-constraint surprise on the next minor

Pre-push verification

`kubectl kustomize kubernetes/infrastructure/ceph-csi-drivers/` builds clean (20 resources)
`kubectl kustomize kubernetes/infrastructure/` (umbrella) builds clean — 144 resources
`kubectl apply -k … --dry-run=server` validates all 20 SAs/RBAC against the live API

Expected recovery sequence after merge

Flux reconciles `infrastructure` KS, applies the 20 new resources
`replicaset-controller` retries pod creation for both ctrlplugin RSes — SAs now exist, pods create
Existing nodeplugin DaemonSet pods stop failing token refresh; new edge-tolerating DaemonSet rolls a pod onto `k8s-edge`
VolumeAttachments for Grafana, Alertmanager, libation, Home Assistant complete; init containers progress
kube-prometheus-stack HelmRelease recovers (Helm rollback failure was downstream of the stalled Grafana Deployment)

Not in this PR

`MON_DISK_LOW` on mons a,b — separate Ceph hygiene issue (mon DB on `/var/lib/rook` host path filling)
Wiki has no v1.20 CSI architecture coverage — worth ingesting once this stabilises

Test plan

Merge to master
Flux source revision flips to new SHA
`kubectl -n rook-ceph get sa` shows the 4 new SAs
`kubectl -n rook-ceph get deploy | grep ctrlplugin` shows 2/2 for both rbd and cephfs
`kubectl -n rook-ceph get ds | grep nodeplugin` shows 4/4 desired/ready for rbd (edge included)
Stuck pods (Grafana, Alertmanager, libation, HA) progress past Init
kube-prometheus-stack HR returns to Ready=True

🤖 Generated with Claude Code

## Summary The rook-ceph HelmRelease's floating `>=1.16.0` constraint silently upgraded to **v1.20.0** about 5 days ago, which extracts CSI driver management into the upstream `ceph-csi-operator` subchart. That subchart's generated `ctrlplugin` Deployments and `nodeplugin` DaemonSets reference per-driver ServiceAccounts (`rbd-ctrlplugin-sa`, `cephfs-ctrlplugin-sa`, `rbd-nodeplugin-sa`, `cephfs-nodeplugin-sa`) that the main `rook-ceph` chart no longer ships — they moved to a separate `ceph-csi-drivers` chart. With the SAs missing, both ctrlplugin Deployments sat at 0/2 with `FailedCreate` for 5 days, blocking PVC attach/detach across the cluster: - `monitoring/kube-prometheus-stack-grafana` — stuck 2d21h on `FailedAttachVolume` - `monitoring/alertmanager-...-0` — same, blocking the kube-prometheus-stack HR (Helm rollback also failed) - `audiobookshelf/libation-...` — same (cron pod, stuck 4d14h) - `home-assistant/home-assistant-...` — pinned to `k8s-edge`, which never got an RBD nodeplugin pod (DaemonSet couldn't create new pods due to missing SA) `monitoring-rules` KS is `dependsOn` blocked by `monitoring`. ## Why static manifests instead of the upstream chart The `ceph-csi-operator/ceph-csi-drivers` chart is the documented Rook v1.20 fix, but it cannot be installed cleanly here: live `helm-controller` is v1.5.4, which predates `disableTakeOwnership`. Helm would refuse to adopt the operator-managed `Driver` and `OperatorConfig` CRs created by the v1.18 → v1.20 auto-migration, and the install would fail. Instead, we render only the SAs + RBAC the operator needs from the chart's templates (v1.0.1) as static manifests under `kubernetes/infrastructure/ceph-csi-drivers/`. The existing operator-managed CRs remain the source of truth. ## Changes | File | Change | |---|---| | `kubernetes/infrastructure/ceph-csi-drivers/serviceaccounts.yaml` | New — 4 SAs | | `kubernetes/infrastructure/ceph-csi-drivers/rbac.yaml` | New — 16 RBAC resources (CR/CRB/R/RB per SA), rendered from ceph-csi-operator/ceph-csi-drivers chart v1.0.1 | | `kubernetes/infrastructure/ceph-csi-drivers/kustomization.yaml` | New | | `kubernetes/infrastructure/kustomization.yaml` | Add \`ceph-csi-drivers/\` to umbrella | | `kubernetes/infrastructure/rook-ceph/helmrelease-operator.yaml` | Pin chart \`>=1.16.0\` → \`~1.20.0\` to prevent the same floating-constraint surprise on the next minor | ## Pre-push verification - [x] \`kubectl kustomize kubernetes/infrastructure/ceph-csi-drivers/\` builds clean (20 resources) - [x] \`kubectl kustomize kubernetes/infrastructure/\` (umbrella) builds clean — 144 resources - [x] \`kubectl apply -k … --dry-run=server\` validates all 20 SAs/RBAC against the live API ## Expected recovery sequence after merge 1. Flux reconciles \`infrastructure\` KS, applies the 20 new resources 2. \`replicaset-controller\` retries pod creation for both ctrlplugin RSes — SAs now exist, pods create 3. Existing nodeplugin DaemonSet pods stop failing token refresh; new edge-tolerating DaemonSet rolls a pod onto \`k8s-edge\` 4. VolumeAttachments for Grafana, Alertmanager, libation, Home Assistant complete; init containers progress 5. kube-prometheus-stack HelmRelease recovers (Helm rollback failure was downstream of the stalled Grafana Deployment) ## Not in this PR - \`MON_DISK_LOW\` on mons a,b — separate Ceph hygiene issue (mon DB on \`/var/lib/rook\` host path filling) - Wiki has no v1.20 CSI architecture coverage — worth ingesting once this stabilises ## Test plan - [ ] Merge to master - [ ] Flux source revision flips to new SHA - [ ] \`kubectl -n rook-ceph get sa\` shows the 4 new SAs - [ ] \`kubectl -n rook-ceph get deploy | grep ctrlplugin\` shows 2/2 for both rbd and cephfs - [ ] \`kubectl -n rook-ceph get ds | grep nodeplugin\` shows 4/4 desired/ready for rbd (edge included) - [ ] Stuck pods (Grafana, Alertmanager, libation, HA) progress past Init - [ ] kube-prometheus-stack HR returns to Ready=True 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-08 18:55:45 +00:00

fix(rook-ceph): ship per-driver CSI ServiceAccounts for v1.20 operator babed07871

The rook-ceph HelmRelease's floating `>=1.16.0` constraint silently
upgraded to v1.20.0, which extracts CSI driver management into the
upstream ceph-csi-operator subchart. That subchart's generated
ctrlplugin Deployments and nodeplugin DaemonSets reference per-driver
ServiceAccounts (`rbd-ctrlplugin-sa`, `cephfs-ctrlplugin-sa`,
`rbd-nodeplugin-sa`, `cephfs-nodeplugin-sa`) that the main rook-ceph
chart no longer ships — they moved to a separate ceph-csi-drivers
chart. With the SAs missing, both ctrlplugin Deployments sat at 0/2
with FailedCreate for 5 days, blocking PVC attach/detach across the
cluster (Grafana, Alertmanager, libation stuck; Home Assistant pinned
to the edge node which never got an RBD nodeplugin pod).

The ceph-csi-drivers chart cannot be installed cleanly here:
helm-controller v1.5.4 predates `disableTakeOwnership`, so Helm would
refuse to adopt the operator-managed `Driver` and `OperatorConfig` CRs
the v1.18 -> v1.20 auto-migration created. Instead, render only the
SAs + RBAC the operator needs from the chart's templates as static
manifests under kubernetes/infrastructure/ceph-csi-drivers/. The
existing operator-managed CRs continue to be the source of truth.

Also pin the rook-ceph chart to `~1.20.0` to prevent the same
floating-constraint surprise on the next minor bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

k.renner merged commit 3ec7baf8eb into master

2026-06-08 18:56:47 +00:00

k.renner referenced this issue from a commit

2026-06-08 18:56:47 +00:00

Merge pull request 'fix(rook-ceph): ship per-driver CSI ServiceAccounts for v1.20 operator' (#12) from fix/rook-ceph-csi-sas into master

claude-bot referenced this pull request

2026-09-25 07:29:40 +00:00

feat(wiki): Flux image automation for the wiki image (plus build.1790319474) #959

claude-bot referenced this issue from a commit

2026-09-25 07:29:42 +00:00

chore(wiki): image build.1790319474 (statement to fact)

Ghost referenced this issue from a commit