homelab/kubernetes/infrastructure/image-automation/README.md

Flux Image Automation

Tracks upstream container images and commits tag bumps to the homelab repo when new versions are published. No bootstrap step — reuses the existing flux-system GitRepository whose Secret was created by flux bootstrap gitea --token-auth and already has write access.

Components

• update-automation.yaml — single ImageUpdateAutomation CR. Walks manifests under ./kubernetes, applies Setters strategy from # {"$imagepolicy": "..."} markers, commits direct to main. Push uses the bootstrap token's identity; commit author is set to Flux Image Automation <security-bot@renner.uno> for visibility.
• paperless-ngx.yaml, etc. — one ImageRepository + ImagePolicy per managed image.

Adding a new managed image

1. Create kubernetes/infrastructure/image-automation/<image-name>.yaml with one ImageRepository (the upstream URL) and one ImagePolicy (the version range to track).
2. Add the file to this directory's kustomization.yaml.
3. In the consumer manifest (kubernetes/apps/<app>/...), append the marker comment to the image line:

   image: ghcr.io/upstream/foo:1.2.3 # {"$imagepolicy": "flux-system:<image-name>"}
   

4. Commit. Flux populates available tags within interval, then ImageUpdateAutomation rewrites the tag to the highest in-policy match on its next pass (~30 min).

Currently enrolled

Image	Range	Notes
paperless-ngx	2.x
audiobookshelf	2.x
music-assistant	2.x
immich-server / immich-machine-learning	2.x	v-prefix filter; bump in lockstep
home-assistant	2026.x	Calver — bump range each year
matter-server	8.x	Pre-release filter
zwave-js-ui	11.x
netbird-ingress	0.x	-rootless suffix filter; pre-1.0, bounded to the 0.x major
wyoming-{whisper,openwakeword,speech-to-phrase}	3.x / 2.x / 1.x
authentik-gate-sync	YYYY-MM-DD-HHMM	In-house (build-ci-images.yml); newest tag wins
kuma-sync	YYYY-MM-DD-HHMM	In-house (build-ci-images.yml); newest tag wins
secret-rotator	YYYY-MM-DD	In-house (build-ci-images.yml); newest tag wins; staged: only the Home Assistant rotation CronJobs carry the marker so far
gatus	5.x	v-prefix filter
velero-plugin-for-aws	1.13.x	Tight: must match velero server major.minor
gotenberg	8.x	Plain-variant filter (rejects -libreoffice/-chromium/-cloudrun/-aws-lambda)
redis-immich	7.4.x	-alpine suffix extract; tight to match LXC 225 compose
redis-paperless	8.x	-alpine suffix extract
ceph	19.x	v-prefix filter; rejects date-suffixed builds
tika	3.x	4-component tags; .0 image-revision pinned in the filter so the extracted key is unique
lidarr	Lidarr 3.x	LinuxServer -ls<n> rebuilds; synthetic <build>.<ls>.0 semver key — major bounded in the filter, not the range
protonmail-bridge	3.x	-1 build-suffix extract
firefox	1.x	-rolling-daily suffix extract; rejects arch-prefixed duplicates
mmwave-configurator	2.x	Plain 3-component filter (rejects addon-/-armv7/-arm64/-amd64 variants)
cnpg-barman-plugin	0.x	v-prefix filter; marker lives in upstream-imported manifest (see below)

Not enrolled (structurally impossible)

• woodpeckerci/woodpecker-cli — the lint image in .woodpecker/ci-manifests-test.yml. It publishes a perfectly good version stream (v3.16.0-alpine), so unlike the three below this is not a tag-format problem. It cannot be enrolled because image-automation can only patch YAML that is a Kubernetes manifest — a maintainer states it plainly in fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the YAML manifest must have apiVersion and kind fields." A Woodpecker pipeline file is plain YAML with neither, so the marker would be silently ignored.

  Note this is not fixable by widening update.path. The path only controls which directory is walked; objects without apiVersion/kind are skipped regardless. Widening the path to ./ buys nothing here and additionally pulls the whole ansible/ and esphome/ trees into the walk, so don't.

  Consequence: the CLI is a manual pin while the server and agent auto-bump, so it will drift. That is a known, accepted gap, covered by the update-audit agent (item 5) rather than by automation. When bumping the server major/minor, bump this too — it must stay on the latest stable like everything else, and a CLI behind the server can pass a config the server then rejects.

Not enrolled (deliberate)

These three images publish no usable version stream — there is nothing for image-policy to sort against:

• fauria/vsftpd — only :latest is published.
• openthread/otbr — only Ubuntu codenames (focal, jammy, hirsute) and :latest/:test/:reference-device; no semver release stream.
• gitea.renner.uno/konairius/ical-proxy — only main, latest, and main-<sha> commit builds; no release tags.

The security-scan agent will continue to surface vuln findings on these as "Skipped / Flux image-automation owns this" even without a policy.

Only a tag-format problem belongs on that list. The policy is to run the latest version and let image-automation hold us there wherever the tags allow it, so an image with a usable semver stream gets enrolled even when a bump carries risk. Where a bad bump is hard to detect, the answer is a note about what to check afterwards — see apps/trmnl/README.md for the panel, whose failure mode is a blank screen rather than a failed rollout — not an exclusion.

Re-import hazard: cnpg-barman-plugin

kubernetes/infrastructure/cnpg-barman-plugin/manifest.yaml is an upstream-generated YAML imported wholesale. The $imagepolicy marker on its image line will get clobbered any time the manifest is re-imported from upstream. If you re-import: re-add the marker on the plugin-barman-cloud image line. The ImagePolicy itself (cnpg-barman-plugin.yaml in this directory) is repo-native and unaffected.

Important

• Strategy is direct push to main, not a PR branch. Bad upstream releases land in main immediately. Mitigation: ImagePolicy ranges must be tight (semver: range: "2.x" not "*") so a major bump requires manual policy widening.
• extract must produce a total order over the tags pattern matches. Any part of the tag that varies has to be either in the extracted key or excluded by the filter. A component that is matched but not extracted makes two different tags share one key, and the winner is then an arbitrary tie-break that can flap and downgrade a running app with no human in the loop. This is not hypothetical: it auto-committed a lidarr downgrade (ls42 → ls11) to master in b858b4aa and rolled the pod. See lidarr.yaml for the two-field case (synthetic semver key) and tika.yaml / protonmail-bridge.yaml for the narrow-the-filter case.
• Image findings still surface in the security-scan agent's report (Skipped / "Flux image-automation owns this") so they remain visible even when an image isn't yet enrolled in automation here.

What watches this

Worth stating explicitly, because for a while nothing did and each plausible reporter deferred to another: the update-audit agent skips $imagepolicy-marked images (ci/agents/update-audit/prompt.md), the security scan records them as "Flux image-automation owns this" (above), and both are correct to — an enrolled image is supposed to be this directory's problem.

What exists today:

• flux-notifications/alerts.yaml emails error-severity events from every image kind. Immediate, but edge-triggered: one mail at the moment of transition, then silence. Nothing says "still broken twelve hours later" and nothing answers "is anything not-Ready right now".
• apps/monitoring-rules/flux-podmonitor.yaml scrapes the controllers, so reconcile latency and controller-level error counts are queryable. Readiness is not on that endpoint: at the controller versions pinned in clusters/homelab/flux-system/gotk-components.yaml, gotk_reconcile_condition and gotk_suspend_status are declared by the shared runtime library and recorded by nothing, so they export no series however the scrape is written.
• apps/monitoring/helmrelease-prometheus.yaml therefore turns on kube-state-metrics customResourceState over eight Flux kinds, which is where per-object readiness now comes from: gotk_resource_info with ready and suspended labels. For ImagePolicy it also carries the resolved tag, and each ImageRepository gets gotk_image_repository_tag_count.
• apps/monitoring-rules/prometheusrule-flux.yaml builds the standing alerts on those series. FluxImageAutomationNotReady fires after an ImageRepository, ImagePolicy or ImageUpdateAutomation has been not-Ready for 2h (two scan intervals for the 31 of 34 repositories at interval: 1h); FluxResourceNotReady covers every other Flux kind at 1h; FluxImagePolicyMetricsMissing is the dead man's switch that says so if the metrics themselves stop. This is the "still broken two hours later" signal the edge-triggered mail above cannot give.

What still does not exist: an alert for a policy that is perfectly Ready while its own filter has frozen it — tika if upstream ships a 3.x. The series for it are now recorded (flux:image_policy_tag_frozen:14d and the two rules under it), but no threshold is attached yet, because none is derivable from the history available today. Measured 2026-09-26 against the 191 automated bump commits and the upstream registries: a flat duration cannot work (gatus went 127 days between bumps while healthy), a 10% tag-count growth comparator fires on three variant-churning repositories and misses tika at 3.0%, relaxing it to one new tag adds velero-plugin-for-aws, whose pin is deliberate, and .status.lastScanResult.latestTags is sorted alphabetically rather than newest-first so it cannot substitute. The recording rules exist to accumulate the per-repository tagCount history that a defensible threshold has to be derived from; the alert is tracked separately.