fix(flux): scope infrastructure health to its operators, not every resource #354

2026-07-22T12:59:37Z

claude-bot commented

2026-07-22 12:59:37 +00:00

Fixes the fragility behind the GitOps freezes this session. The infrastructure Kustomization used wait: true with no healthChecks — and with wait: true, Flux ignores healthChecks and blocks Ready on every resource it applies (hundreds). Since ~28 kustomizations dependsOn it, one crashlooping workload froze the entire GitOps apply chain.

What happened

A crowdsec-agent crashloop (0/3, "InProgress") held all of GitOps hostage for hours — merges didn't go live — even though crowdsec is a non-critical app nothing depends on for startup. The 15m health-check timeout looped every 10m, so infrastructure never reported Ready and every dependent stayed blocked. Every context deadline exceeded on a reconcile this session was the health check waiting on crowdsec. (The Kustomization itself reconciles in ~1.3s — it was never slow; it was blocked.)

The fix

wait: false + explicit healthChecks covering only the operators/providers dependents actually consume:

resource	why
cilium	CNI — nothing networks without it
cert-manager	Certificates → TLS Secrets apps mount
external-secrets + openbao	the ESO backend ExternalSecrets resolve against
rook-ceph-operator	storage — apps mount PVCs
cloudnative-pg	DB operator — seed jobs create databases

Everything else still applies — it just no longer gates readiness, so a non-critical crashloop (crowdsec, monitoring, an app) can't cascade across the cluster. This mirrors what infrastructure-ceph-cluster already does with healthCheckExprs for Rook CRs.

Verified live

Applied the change: infrastructure reconciles Ready=True with wait=false on the 6 checks, and all dependent kustomizations stayed Ready (0 not-ready). The 6 operators are the complete set of what infrastructure-crds (ClusterIssuer/ClusterSecretStore need cert-manager/ESO), -cnpg-barman, -ceph-cluster, and the apps (certs/ESO/storage/CNI/DB) require.

🤖 Generated with Claude Code

Fixes the fragility behind the GitOps freezes this session. The `infrastructure` Kustomization used `wait: true` with no `healthChecks` — and with `wait: true`, Flux **ignores healthChecks and blocks Ready on every resource it applies** (hundreds). Since ~28 kustomizations `dependsOn` it, **one crashlooping workload froze the entire GitOps apply chain**. ## What happened A `crowdsec-agent` crashloop (0/3, "InProgress") held all of GitOps hostage for hours — merges didn't go live — even though crowdsec is a non-critical app nothing depends on for startup. The 15m health-check timeout looped every 10m, so `infrastructure` never reported Ready and every dependent stayed blocked. Every `context deadline exceeded` on a reconcile this session was the health check waiting on crowdsec. (The Kustomization itself reconciles in ~1.3s — it was never slow; it was *blocked*.) ## The fix `wait: false` + explicit `healthChecks` covering only the operators/providers dependents actually consume: | resource | why | |---|---| | cilium | CNI — nothing networks without it | | cert-manager | Certificates → TLS Secrets apps mount | | external-secrets + openbao | the ESO backend ExternalSecrets resolve against | | rook-ceph-operator | storage — apps mount PVCs | | cloudnative-pg | DB operator — seed jobs create databases | Everything else still **applies** — it just no longer **gates readiness**, so a non-critical crashloop (crowdsec, monitoring, an app) can't cascade across the cluster. This mirrors what `infrastructure-ceph-cluster` already does with `healthCheckExprs` for Rook CRs. ## Verified live Applied the change: `infrastructure` reconciles **Ready=True with wait=false** on the 6 checks, and **all dependent kustomizations stayed Ready** (0 not-ready). The 6 operators are the complete set of what `infrastructure-crds` (ClusterIssuer/ClusterSecretStore need cert-manager/ESO), `-cnpg-barman`, `-ceph-cluster`, and the apps (certs/ESO/storage/CNI/DB) require. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-22 12:59:37 +00:00

fix(flux): scope infrastructure health to its operators, not every resource 93944ff525

The infrastructure Kustomization used wait: true with no healthChecks. With
wait: true Flux ignores healthChecks and blocks the Kustomization's Ready on
EVERY resource it applies (hundreds). Because ~28 kustomizations dependsOn
infrastructure, one crashlooping workload froze the entire GitOps apply chain:
a crowdsec-agent crashloop held all of GitOps hostage for hours — merges didn't
go live — even though crowdsec is a non-critical app nothing depends on for
startup. The 15m health-check timeout looped every 10m, so infrastructure never
reported Ready and every dependent stayed blocked.

Switch to wait: false + explicit healthChecks covering only the operators and
providers dependents actually consume: cilium (CNI), cert-manager (TLS Secrets),
external-secrets + openbao (the ESO backend ExternalSecrets resolve against),
rook-ceph-operator (storage), cloudnative-pg (DB operator for seed jobs).
Everything else still applies — it just no longer gates readiness, so a
non-critical crashloop can't cascade across the cluster. This mirrors the
pattern infrastructure-ceph-cluster already uses (healthCheckExprs for Rook).

Verified live: infrastructure reconciles Ready on the 6 checks (wait=false),
and all dependent kustomizations stayed Ready.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-07-22 13:19:48 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 93944ff5255e2cb0e6d6f6736259c7591bf9fbac.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes a cascading-failure vulnerability where a single non-critical crashlooping workload (crowdsec-agent) could freeze the entire GitOps pipeline by changing the infrastructure Kustomization from wait: true (which blocks on all resources) to wait: false with explicit healthChecks for only the 6 critical operators (CNI, cert issuance, secrets, storage, DB). The change is minimal, well-commented, and addresses a real production issue.

🟢 Nits

Line 16: "ignores healthChecks and blocks ... on EVERY resource it applies (hundreds)" — the comment could clarify that this is specific behavior of wait: true without explicit healthChecks, and that healthCheckExprs (used in infrastructure-ceph-cluster) is a different mechanism that works with wait: true. Minor clarity point only.

Tests

No tests to verify here (this is Kubernetes cluster configuration, not code). The PR body documents live verification: the author applied the change and confirmed infrastructure reconciles Ready with only the 6 health checks passing, and all dependent Kustomizations remained Ready. This is appropriate validation for infrastructure-as-code.

Verification

All 6 health checks reference existing HelmReleases in correct namespaces:

cilium (kube-system) — network layer ✓
cert-manager (cert-manager) — certificate authority ✓
external-secrets (external-secrets) — secret management ✓
openbao (openbao) — ESO backend ✓
rook-ceph-operator (rook-ceph) — persistent storage ✓
cloudnative-pg (database) — database operator ✓

The selected set is well-justified: these are the transitive dependencies that downstream Kustomizations (infrastructure-crds, infrastructure-cnpg-barman, infrastructure-ceph-cluster) and applications actually require. Non-critical operators (crowdsec, traefik, velero, etc.) are intentionally excluded so their failures don't cascade. The approach mirrors the healthCheckExprs pattern already used in infrastructure-ceph-cluster.

Recommendation

Looks reasonable — no blocking issues; the nit above is optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `93944ff5255e2cb0e6d6f6736259c7591bf9fbac`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes a cascading-failure vulnerability where a single non-critical crashlooping workload (crowdsec-agent) could freeze the entire GitOps pipeline by changing the `infrastructure` Kustomization from `wait: true` (which blocks on all resources) to `wait: false` with explicit healthChecks for only the 6 critical operators (CNI, cert issuance, secrets, storage, DB). The change is minimal, well-commented, and addresses a real production issue. ## 🟢 Nits - Line 16: "ignores healthChecks and blocks ... on EVERY resource it applies (hundreds)" — the comment could clarify that this is specific behavior of `wait: true` without explicit healthChecks, and that `healthCheckExprs` (used in infrastructure-ceph-cluster) is a different mechanism that works with `wait: true`. Minor clarity point only. ## Tests No tests to verify here (this is Kubernetes cluster configuration, not code). The PR body documents live verification: the author applied the change and confirmed `infrastructure` reconciles Ready with only the 6 health checks passing, and all dependent Kustomizations remained Ready. This is appropriate validation for infrastructure-as-code. ## Verification All 6 health checks reference existing HelmReleases in correct namespaces: - cilium (kube-system) — network layer ✓ - cert-manager (cert-manager) — certificate authority ✓ - external-secrets (external-secrets) — secret management ✓ - openbao (openbao) — ESO backend ✓ - rook-ceph-operator (rook-ceph) — persistent storage ✓ - cloudnative-pg (database) — database operator ✓ The selected set is well-justified: these are the transitive dependencies that downstream Kustomizations (infrastructure-crds, infrastructure-cnpg-barman, infrastructure-ceph-cluster) and applications actually require. Non-critical operators (crowdsec, traefik, velero, etc.) are intentionally excluded so their failures don't cascade. The approach mirrors the `healthCheckExprs` pattern already used in infrastructure-ceph-cluster. ## Recommendation Looks reasonable — no blocking issues; the nit above is optional.

k.renner merged commit f14fb80589 into master

2026-07-22 13:26:25 +00:00

k.renner referenced this issue from a commit

2026-07-22 13:26:27 +00:00

Merge pull request 'fix(flux): scope infrastructure health to its operators, not every resource' (#354) from fix/infra-healthchecks into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#354