konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Merge pull request 'ci(monitoring-rules): machine-check PromQL with promtool check rules + test rules (REN-161)' (#1026) from atlas/ren-161-promtool-ci into master
ci/woodpecker/cron/issue-fix Pipeline was successful
Details
ci/woodpecker/cron/alert-monitor Pipeline was successful
Details

Browse Source
This commit was merged in pull request #1026.
This commit is contained in:
argus-bot (Paperclip service account)
2026-09-26 08:32:25 +00:00
parent bc1c69d356 9a3c6ef9af
commit 2608c264d3
8 changed files with 2026 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.woodpecker/ci-manifests-test.yml
+77 -15
View File
@@ -17,30 +17,43 @@
# here is a static check that either passes or fails on the checkout alone, so it
# can gate PRs without becoming a bottleneck or a flake source.
when:
# Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
# step on one already-cached image, so running it everywhere is free; this one
# pulls three images and builds every kustomization in the tree, which is real
# Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is two
# steps on already-cached images, so running it everywhere is free; this one
# pulls four images and builds every kustomization in the tree, which is real
# cost to put on a docs-only PR. The filter covers everything the steps actually read:
# kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
# so an edit to this file re-runs its own checks).
# kubernetes/** for the manifests, scripts and promtool suites, .woodpecker/**
# for the lint (and so an edit to this file re-runs its own checks), and the
# promtool driver script by name.
#
# That last entry is NOT covered by kubernetes/**: promtool-rules.sh lives
# under scripts/, so without it a PR that only edits the script would never run
# the check it drives. It is listed as one file rather than scripts/ci/**
# because the rest of that directory is CI-agent helpers with nothing to do
# with this pipeline, and pulling four images for them is the cost this filter
# exists to avoid. Their own tests run in ci-scripts-test.yml on every PR.
- event: pull_request
path:
- "kubernetes/**"
- ".woodpecker/**"
- "scripts/ci/promtool-rules.sh"
# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
# the same shape — `find`, a floor guard against empty discovery, then a
# file-fed `while read`. Two copies is deliberate: the two traps being commented
# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
# where `fail` cannot propagate) are different, and both are worth spelling out
# where they bite. A third instance is the point to stop copying and factor it
# into a shared script under `scripts/ci/` — which is already covered by
# ci-scripts-test.yml, so it would gain unit tests in the process.
# THE THIRD DISCOVERY CHECK WENT TO A SCRIPT. kustomize-build and shellcheck
# both hand-roll the same shape — `find`, a floor guard against empty discovery,
# then a file-fed `while read`. Two copies was deliberate: the two traps being
# commented around (BusyBox find lacking `-printf`, and a piped loop running in
# a subshell where `fail` cannot propagate) are different, and both are worth
# spelling out where they bite. An earlier revision of this comment said a third
# instance would be the point to stop copying and factor it into `scripts/ci/`,
# where it would gain unit tests — and that is what `promtool` below does. Its
# discovery, floors and guards live in scripts/ci/promtool-rules.sh and are
# tested by scripts/ci/test-promtool-rules.sh on every PR via
# ci-scripts-test.yml. A FOURTH inline copy should go the same way; do not add
# one here.
#
# Every step carries `depends_on: []` so all four run in PARALLEL and every
# Every step carries `depends_on: []` so all five run in PARALLEL and every
# failure shows in one run. Woodpecker's default is sequential, which would stop
# at the first failure and make a contributor fix-and-repush to discover the
# next — and these four are independent, cheap, offline checks with no ordering
# next — and these five are independent, cheap, offline checks with no ordering
# between them. Keep it on every step: `depends_on` is per-step, so dropping it
# from one silently re-serialises that step behind the others.
steps:
@@ -258,3 +271,52 @@ steps:
exit 1
fi
echo "ok no digest pins under openbao/rotation/"
# Machine-check the alert rules: parse every PrometheusRule, run the committed
# promtool unit suites, then damage the rules on purpose and require the suites
# to notice. Everything it does lives in scripts/ci/promtool-rules.sh; read
# that file and kubernetes/apps/monitoring-rules/tests/README.md for the why.
#
# The gap it closes: the other four steps prove a PrometheusRule assembles into
# valid YAML and nothing more — not that an expression parses, not that a label
# join matches anything, not that a `for` window does what its comment claims.
# On REN-129 the CNPG surge rule was written `for: 2m`, which fires on six of
# seven measured nights and would have mailed the owner twice a night at 04:16
# and 04:22 local for a backlog that clears itself. Nothing here would have
# caught it, and the arithmetic that did catch it was hand-written from scratch
# twice, in two runs, and thrown away both times because there was nowhere in
# the repo to commit it. It is now mutations.txt:surge-for-2m.
#
# Offline, like the rest of this pipeline: no cluster, no registry, no secrets,
# no network. promtool reads files and exits.
#
# IMAGE — prom/prometheus, not a registry-local ci-* image, and pinned to an
# exact upstream version rather than a dated tag. The dated-tag convention
# guarded by no-digest-pins above is about OUR images, whose digests are
# garbage-collected out from under a pin; this is upstream and immutable.
#
# The version must be 3.x, and that is not a preference. The cluster runs
# whatever kube-prometheus-stack >= 72.0.0 ships, which is Prometheus 3, and
# alert `for` evaluation is version-sensitive — a 2.x promtool would be
# checking a different evaluator than production, so the margins the suites
# assert would be claims about the wrong program. promtool-rules.sh asserts the
# major version it was handed and fails on 2.x, so a careless bump here is a
# red step rather than a quietly wrong check. Keep this at the latest stable
# 3.x, per the standing rule that any software uses the latest stable version.
#
# This image is NOT covered by image-automation (same reason as
# woodpecker-lint above: a Woodpecker pipeline file has no apiVersion/kind, so
# an $imagepolicy marker here is silently ignored — fluxcd/flux2#2826). The
# bump is manual.
#
# `prom/prometheus` is busybox-based and ships /bin/sh, so it can run a
# `commands:` block — unlike the plain woodpecker-cli tag, which cannot. Its
# find/grep/awk/sed are BusyBox, which is why promtool-rules.sh avoids
# `find -printf` and `grep -r --include`; it was verified against BusyBox
# 1.35.0 as well as GNU. The image also declares `USER nobody`, so the script
# writes its stripped mirror under mktemp and never into the workspace.
- name: promtool
depends_on: [] # see note above `steps:`
image: prom/prometheus:v3.15.0
commands:
- sh scripts/ci/promtool-rules.sh
.woodpecker/ci-scripts-test.yml
+27 -3
View File
@@ -8,15 +8,39 @@
# against adversarial PR input. Running it here fails the offending PR's checks
# before it can merge.
#
# scripts/ci/test-promtool-rules.sh is the same idea for the alert-rule gate.
# What it tests is not whether promtool evaluates PromQL — ci-manifests-test.yml
# does that with the real binary against the real rules — but whether
# promtool-rules.sh REFUSES TO REPORT SUCCESS when it has checked nothing: zero
# manifests discovered, zero suites discovered, a `rule_files` path that does not
# resolve (promtool warns, loads zero rules, and passes every assertion), a
# mutation the suites no longer catch. Those are the states a green pipeline
# cannot tell apart from real coverage, so they need a test that asserts the
# failure. It stubs promtool, so it needs neither the binary nor the heavy image.
#
# Both are unfiltered by path on purpose: this pipeline is two steps on one
# already-cached image, so running it everywhere is free, and a guard whose test
# only runs when someone remembers to touch the right directory is not a guard.
#
# Only in homelab (scripts/ci/ lives here); the file is not deployed to the
# mirrored repos, so it does not run on their PRs. The test is offline (no token,
# no network, no cluster) — the ci-claude image just provides sh + jq, and it is
# already pulled by pr-review.yml on the same event, so the runner has it cached.
# mirrored repos, so it does not run on their PRs. Both tests are offline (no
# token, no network, no cluster) — the ci-claude image just provides sh + jq, and
# it is already pulled by pr-review.yml on the same event, so the runner has it
# cached.
when:
- event: pull_request
# `depends_on: []` on both so they run in parallel and one failure does not hide
# the other. Same reasoning as the note above `steps:` in ci-manifests-test.yml.
steps:
- name: model-selection-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
commands:
- sh scripts/ci/test-model-selection.sh
- name: promtool-rules-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
commands:
- sh scripts/ci/test-promtool-rules.sh
kubernetes/apps/monitoring-rules/tests/README.md
+125
View File
@@ -0,0 +1,125 @@
# promtool unit suites for the PrometheusRules in this directory
`promtool test rules` suites, run on every pull request that touches
`kubernetes/**` by the `promtool` step in
[`.woodpecker/ci-manifests-test.yml`](../../../../.woodpecker/ci-manifests-test.yml)
via [`scripts/ci/promtool-rules.sh`](../../../../scripts/ci/promtool-rules.sh).
## Why this exists
Before this, CI proved that every `PrometheusRule` in the tree assembled into
valid YAML and nothing else. It did not check that an expression parses, that a
label join matches anything, or that a `for` window behaves the way its comment
claims.
That was not theoretical. On [REN-129](/REN/issues/REN-129) the surge rule in
`prometheusrule-cnpg-backup.yaml` was written `for: 2m` and reviewed at that
value. Against Iris's seven measured nights it fires on six of them — two owner
emails a night, 04:16 and 04:22 local, for a WAL backlog that clears itself in
under ten minutes. It was caught by a human writing the arithmetic out by hand,
twice, in two separate runs, and throwing the work away both times because
there was nowhere in the repo to put it. That rule is now
`mutations.txt:surge-for-2m`.
## Layout
| File | What it is |
| --- | --- |
| `cnpg-backup_test.yaml` | The suite. **Generated** — see below. |
| `generate-cnpg-backup-test.py` | Regenerates it from the REN-129 measurement table. |
| `mutations.txt` | Negative controls. Each must make the suite fail. |
## Adding a suite
Name it `<something>_test.yaml` in this directory. Discovery is by that suffix,
and `promtool-rules.sh` fails if it finds none, so a suite in the wrong place
is a red pipeline rather than a silent gap.
**`rule_files` entries are relative to this directory, not the repo root.**
promtool joins `dirname(suite)` onto each entry, so a manifest in the parent
directory is `../name.yaml`. Getting this wrong is not a test failure: promtool
prints `WARNING: no file match pattern …`, loads **zero rules**, and every
`exp_alerts: []` assertion then passes against an empty rule set. The first
draft of the suite here shipped exactly that way — ninety silence assertions,
green, testing nothing. `promtool-rules.sh` resolves every entry itself before
running promtool and fails on a miss, because promtool has no flag that makes
it fatal.
`promtool` cannot read a `PrometheusRule` directly: it wants a bare `groups:`
document, and a `PrometheusRule` carries that under `spec:`. The script strips
`spec:` into a temporary mirrored tree with the same relative layout and runs
there, so `../name.yaml` resolves in the mirror exactly as it reads here.
Nothing is written back into the checkout.
## Regenerating `cnpg-backup_test.yaml`
```sh
python3 kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py \
> kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml
```
and commit the result. `--report` prints the reconstruction self-check table
instead of the suite.
The generator exists so the numbers in the suite are auditable: the seven
measured nights are rebuilt from the `(peak, peak minute, fill, drain)` tuples
Iris published on REN-129, not hand-typed, so a reader can check them against
the measurement rather than taking them on trust. It is not a hermetic
artifact, and **CI does not regenerate and diff it** — the promtool image has
no Python, and byte-equality with a generator is not what stops a suite from
being weakened. The mutation phase is.
Two things the generator refuses to emit, both because they happened:
- a window that does not cover the bursts. An earlier revision had `T0` at
`8 * 60 + 8` (08:08Z) instead of `2 * 60 + 8`, seven hours after every burst,
so all seven nights rebuilt as forty zeroes and their silence assertions
passed without touching a threshold. `assert_window_covers_bursts()` now
fails instead.
- a night that clears the threshold *sooner* than the measured one, which would
be a weaker silence test than the night it claims to stand for. Longer is
fine and expected — a linear decay at the published mean drain rate holds the
tail above 5 a minute or two longer than the real trace on four of the seven
nights, which makes the suite harder to pass, not easier.
## Negative controls
A suite that passes proves the rules match the assertions. It does not prove
the assertions reach the rules — both failure modes above were green. So
`promtool-rules.sh` runs a second phase: for each entry in `mutations.txt` it
breaks one value the rules depend on and requires the suites to **fail**. A
mutation nothing notices is reported as a hole in the suite.
This is also the answer to "why is that case in there". `surge-for-5m` is
silent on all seven real nights and fires only on the adversarial envelope
case, so deleting the envelope case makes `for: 8m` reducible to `5m` with the
suite still green. The control is what makes the case load-bearing.
Mutations are `name|manifest-path|sed-program`, applied to the stripped copy in
the mirror (two spaces shallower than the manifest on disk — anchor on
`^[[:space:]]*`). A program that matches nothing is a failure, not a skip:
otherwise renaming a rule quietly turns its control into a no-op.
## Reading `for` margins
Margin is **pending** time, which is one minute less than the run length. An
alert goes pending at the first evaluation where the expression is true and
fires when `now - activeAt >= for`, so N consecutive one-minute samples above a
threshold accumulate N-1 minutes. A trace with 5 samples above 64 does not fire
`for: 5m`; it fires `for: 4m`. The `>5 pend` and `>64 pend` columns in
`--report` are the numbers to compare a `for` against — not the sample counts
next to them.
## Running it locally
```sh
# promtool must be 3.x: the cluster runs kube-prometheus-stack >= 72.0.0, and
# alert `for` evaluation is version-sensitive, so a 2.x promtool would be
# checking a different evaluator than production. The script enforces this.
sh scripts/ci/promtool-rules.sh
```
The script's own guards — the discovery floors, the `rule_files` resolution, the
mutation phase — are tested offline and without promtool by
`scripts/ci/test-promtool-rules.sh`, which runs on every pull request via
`.woodpecker/ci-scripts-test.yml`.
kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml
+592
View File
@@ -0,0 +1,592 @@
# GENERATED by generate-cnpg-backup-test.py — see README.md in this
# directory before editing by hand.
#
# Unit tests for kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml.
# Every assertion below is either one of Iris's seven measured nights
# (REN-129) rebuilt from its published (peak, fill, drain) numbers, an
# adversarial combination of the extremes of those numbers, or a failure
# this cluster has not had but the rules exist to catch.
#
# `rule_files` entries are resolved RELATIVE TO THIS FILE'S DIRECTORY, not
# to the working directory promtool was invoked from — promtool joins
# dirname(testfile) onto each entry. So this is `../`, and it resolves in
# the mirrored tree scripts/ci/promtool-rules.sh builds, where the
# `spec:`-stripped manifests sit in the same layout as the real ones.
#
# A path that does NOT resolve is only a WARNING to promtool: it loads zero
# rules, every exp_alerts: [] assertion passes against nothing, and the
# suite exits 0. promtool-rules.sh resolves every entry itself before
# running and fails on a miss; that check is the only thing standing
# between a typo here and a green pipeline that tests nothing.
rule_files:
- ../prometheusrule-cnpg-backup.yaml
evaluation_interval: 1m
tests:
# ---------------------------------------------------------------------
# 1-7. THE SEVEN MEASURED NIGHTS. Both rules must stay silent through
# every one of them. This is the half of the suite that has already
# caught something: the surge rule shipped for review at `for: 2m` and
# fires here on 6 of these 7 nights (mutations.txt:surge-for-2m).
# ---------------------------------------------------------------------
# 2026-09-20: the nightly Home Assistant recorder purge at 04:12 local.
# peak 106 at 02:15Z, fill 35.3/min, drain 13.2/min -> 10 samples above 5 (9m pending vs for: 15m), 5 above 64 (4m pending vs for: 8m).
- name: nightly burst 2026-09-20 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 0 35 71 106 93 80 66 53 40 27 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 106
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 106
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:18Z, backlog 66
- eval_time: 10m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:18Z, backlog 66
- eval_time: 10m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:22Z, backlog 14
- eval_time: 14m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:22Z, backlog 14
- eval_time: 14m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:24Z, backlog 0
- eval_time: 16m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:24Z, backlog 0
- eval_time: 16m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 2026-09-21: the nightly Home Assistant recorder purge at 04:12 local.
# peak 100 at 02:15Z, fill 33.3/min, drain 14.3/min -> 9 samples above 5 (8m pending vs for: 15m), 4 above 64 (3m pending vs for: 8m).
- name: nightly burst 2026-09-21 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 0 33 67 100 86 71 57 43 28 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 100
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 100
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:17Z, backlog 71
- eval_time: 9m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:17Z, backlog 71
- eval_time: 9m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:21Z, backlog 14
- eval_time: 13m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:21Z, backlog 14
- eval_time: 13m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:23Z, backlog 0
- eval_time: 15m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:23Z, backlog 0
- eval_time: 15m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 2026-09-22: the nightly Home Assistant recorder purge at 04:12 local.
# peak 101 at 02:15Z, fill 33.7/min, drain 14.4/min -> 9 samples above 5 (8m pending vs for: 15m), 4 above 64 (3m pending vs for: 8m).
- name: nightly burst 2026-09-22 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 0 34 67 101 87 72 58 43 29 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 101
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 101
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:17Z, backlog 72
- eval_time: 9m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:17Z, backlog 72
- eval_time: 9m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:21Z, backlog 15
- eval_time: 13m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:21Z, backlog 15
- eval_time: 13m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:23Z, backlog 0
- eval_time: 15m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:23Z, backlog 0
- eval_time: 15m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 2026-09-23: the nightly Home Assistant recorder purge at 04:12 local.
# peak 119 at 02:15Z, fill 39.7/min, drain 14.9/min -> 10 samples above 5 (9m pending vs for: 15m), 5 above 64 (4m pending vs for: 8m).
- name: nightly burst 2026-09-23 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 0 40 79 119 104 89 74 59 44 30 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 119
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 119
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:18Z, backlog 74
- eval_time: 10m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:18Z, backlog 74
- eval_time: 10m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:22Z, backlog 15
- eval_time: 14m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:22Z, backlog 15
- eval_time: 14m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:24Z, backlog 0
- eval_time: 16m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:24Z, backlog 0
- eval_time: 16m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 2026-09-24: the nightly Home Assistant recorder purge at 04:12 local.
# peak 56 at 02:14Z, fill 18.7/min, drain 14.0/min -> 6 samples above 5 (5m pending vs for: 15m), 0 above 64 (0m pending vs for: 8m).
- name: nightly burst 2026-09-24 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 19 37 56 42 28 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:14Z, backlog 56
- eval_time: 6m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:14Z, backlog 56
- eval_time: 6m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:17Z, backlog 14
- eval_time: 9m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:17Z, backlog 14
- eval_time: 9m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:19Z, backlog 0
- eval_time: 11m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:19Z, backlog 0
- eval_time: 11m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 2026-09-25: the nightly Home Assistant recorder purge at 04:12 local.
# peak 91 at 02:15Z, fill 22.8/min, drain 13.0/min -> 10 samples above 5 (9m pending vs for: 15m), 4 above 64 (3m pending vs for: 8m).
- name: nightly burst 2026-09-25 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 23 46 68 91 78 65 52 39 26 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 91
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 91
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:17Z, backlog 65
- eval_time: 9m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:17Z, backlog 65
- eval_time: 9m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:21Z, backlog 13
- eval_time: 13m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:21Z, backlog 13
- eval_time: 13m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:23Z, backlog 0
- eval_time: 15m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:23Z, backlog 0
- eval_time: 15m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 2026-09-26: the nightly Home Assistant recorder purge at 04:12 local.
# peak 113 at 02:15Z, fill 28.5/min, drain 14.1/min -> 11 samples above 5 (10m pending vs for: 15m), 5 above 64 (4m pending vs for: 8m).
- name: nightly burst 2026-09-26 — recorder purge, benign, self-clearing
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 28 57 86 113 99 85 71 57 42 28 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 113
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 113
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:18Z, backlog 71
- eval_time: 10m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:18Z, backlog 71
- eval_time: 10m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:22Z, backlog 14
- eval_time: 14m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:22Z, backlog 14
- eval_time: 14m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:24Z, backlog 0
- eval_time: 16m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:24Z, backlog 0
- eval_time: 16m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# ---------------------------------------------------------------------
# 8. THE ADVERSARIAL ENVELOPE. A night that did not happen, built from
# the worst fill (39.7/min, 09-23), the highest peak (119, 09-23) and
# the slowest drain (13.0/min, 09-25) in the week. Inside what the
# cluster has demonstrated, worse than anything it actually did.
#
# This is where the two `for` windows get measured for margin rather
# than asserted about. Margin is PENDING time, which is one less than
# the run length — an alert goes pending on the first true evaluation
# and fires at activeAt + for, so 6 consecutive samples above 64
# accumulate 5 minutes, not 6:
#
# above 5: 11 samples -> 10m pending vs `for: 15m` -> 5m margin
# above 64: 6 samples -> 5m pending vs `for: 8m` -> 3m margin
#
# The worst MEASURED night accumulates 4m above 64 (5 samples, 09-20
# and 09-23), so the envelope is one minute tighter than anything that
# actually happened. 3 minutes is the number to defend if anyone
# proposes lowering `for: 8m` — and mutations.txt:surge-for-5m is the
# control that makes this case load-bearing rather than decorative: at
# `for: 5m` the surge rule is still silent on all seven real nights and
# fires only here. Delete this case and that mutation stops being
# caught.
# ---------------------------------------------------------------------
# Synthetic: max fill 39.7 + max peak 119 + min drain 13.0.
# peak 119 at 02:15Z, fill 39.7/min, drain 13.0/min -> 11 samples above 5 (10m pending vs for: 15m), 6 above 64 (5m pending vs for: 8m).
- name: adversarial envelope — worst measured fill and peak, slowest drain
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# 02:08Z .. 02:47Z, one sample per minute
values: '0 0 0 0 0 40 79 119 106 93 80 67 54 41 28 15 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# The cumulative archived counter on the same metric name. The
# rules must never read it; see mutations.txt:no-value-selector.
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
# 02:15Z, backlog 119
- eval_time: 7m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:15Z, backlog 119
- eval_time: 7m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:19Z, backlog 67
- eval_time: 11m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:19Z, backlog 67
- eval_time: 11m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:23Z, backlog 15
- eval_time: 15m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:23Z, backlog 15
- eval_time: 15m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# 02:26Z, backlog 0
- eval_time: 18m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
# 02:26Z, backlog 0
- eval_time: 18m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# ---------------------------------------------------------------------
# 9. THE FAILURE BOTH RULES EXIST FOR: an archiver that stopped with
# 70 segments (1.1 GiB) queued. Not a burst — it does not drain.
# Surge fires first (8m after the backlog crosses 64), the
# load-bearing rule 7 minutes later (15m after it crosses 5). Both
# assertions bracket the transition: one minute early nothing is
# firing, on the minute it is.
# ---------------------------------------------------------------------
- name: stalled archiver at 70 segments — both rules must fire
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
values: '0 0 0 0 0 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
# Flat, because a stalled archiver archives nothing. The rules do
# not read this; it is here so the input stays realistic.
values: '148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000'
alert_rule_test:
# Backlog crosses 64 at t=5m, so `for: 8m` elapses at t=13m.
- eval_time: 12m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
- eval_time: 13m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts:
# No `role` label: proof that `max by (namespace, pod)`
# aggregated it away. See mutations.txt:no-max-by.
- exp_labels:
severity: warning
namespace: home-assistant
pod: ha-pg-1
exp_annotations:
summary: "CNPG WAL archiving cannot keep up on home-assistant/ha-pg-1"
description: "70 WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129)."
# Backlog crosses 5 at t=5m, so `for: 15m` elapses at t=20m.
- eval_time: 19m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
- eval_time: 20m
alertname: CNPGWalArchiveBacklog
exp_alerts:
- exp_labels:
severity: warning
namespace: home-assistant
pod: ha-pg-1
exp_annotations:
summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
description: "70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
# ---------------------------------------------------------------------
# 10. THE STALL THE SURGE RULE CANNOT SEE: 12 segments, stuck. Under
# 1 GiB forever, so the surge rule is right to stay silent — and this
# is exactly the shape that makes the >5 rule load-bearing rather than
# a slower copy of it. Point-in-time recovery is just as broken at 12
# segments as at 70.
# ---------------------------------------------------------------------
- name: low stall at 12 segments — only the >5 rule may fire
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
values: '0 0 0 0 0 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12'
alert_rule_test:
- eval_time: 20m
alertname: CNPGWalArchiveBacklog
exp_alerts:
- exp_labels:
severity: warning
namespace: home-assistant
pod: ha-pg-1
exp_annotations:
summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
description: "12 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
# 34 minutes at 12 segments and the surge rule is still quiet.
- eval_time: 39m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# ---------------------------------------------------------------------
# 11. PINNED AT EXACTLY 5 for the whole window. `> 5` is strict, so
# this is silence. A rule written `>= 5` would page on it, and a rule
# that had lost its `value="ready"` selector would page on the `done`
# counter regardless of what `ready` does — which is what the second
# series here is for.
# ---------------------------------------------------------------------
- name: pinned at exactly 5 — strict > means silence
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
values: '5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
alert_rule_test:
- eval_time: 39m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
- eval_time: 39m
alertname: CNPGWalArchiveBacklogSurge
exp_alerts: []
# ---------------------------------------------------------------------
# 12. FAILOVER DURING A STALL — the case `max by (namespace, pod)` was
# added for. The backlog starts at t=5m on the primary; at t=11m the
# instance is promoted and `role` changes, which is a NEW SERIES to
# Prometheus and restarts the `for` timer on the raw expression.
# Aggregating `role` away keeps one continuous series, so the page
# still lands 15m after the backlog started (t=20m) rather than 15m
# after the failover. Without `max by` this case is silent at t=20m —
# see mutations.txt:no-max-by.
# ---------------------------------------------------------------------
- name: role flip mid-stall — for timer must not restart
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
# t=0..10m, then this series ends.
values: '0 0 0 0 0 70 70 70 70 70 70'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="replica",value="ready"}'
# Starts at t=11m. `_` is a gap, not a zero.
values: '_ _ _ _ _ _ _ _ _ _ _ 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70'
alert_rule_test:
- eval_time: 19m
alertname: CNPGWalArchiveBacklog
exp_alerts: []
- eval_time: 20m
alertname: CNPGWalArchiveBacklog
exp_alerts:
- exp_labels:
severity: warning
namespace: home-assistant
pod: ha-pg-1
exp_annotations:
summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
description: "70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
# ---------------------------------------------------------------------
# 13. ONE STALLED INSTANCE AMONG HEALTHY ONES. There are 12 CNPG
# instances in this cluster. `by (namespace, pod)` has to keep them
# apart: exactly one alert, naming the pod that is actually stuck, so
# the 04:00 reader is not sent to look at the wrong Postgres.
# ---------------------------------------------------------------------
- name: one stalled pod among healthy ones — exactly one alert
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
values: '0 0 0 0 0 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-2",container="postgres",role="replica",value="ready"}'
values: '0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
- series: 'cnpg_collector_pg_wal_archive_status{namespace="immich",pod="immich-pg-1",container="postgres",role="primary",value="ready"}'
values: '0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
alert_rule_test:
- eval_time: 20m
alertname: CNPGWalArchiveBacklog
exp_alerts:
- exp_labels:
severity: warning
namespace: home-assistant
pod: ha-pg-1
exp_annotations:
summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
description: "70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
# ---------------------------------------------------------------------
# 14. THE EPOCH-ZERO GUARD on CNPGBackupFailed. The metric is a unix
# timestamp and reads 0 when no failure was ever recorded, so the rule
# carries a `> 0` guard to stop that default being read as "failed in
# 1970". Held at 0 for the whole window: silence. Nothing else in this
# suite exercises that rule, and a dropped guard would page on every
# instance in the cluster forever.
# ---------------------------------------------------------------------
- name: never-failed backup reads timestamp 0 — must not fire
interval: 1m
input_series:
- series: 'cnpg_collector_last_failed_backup_timestamp{namespace="home-assistant",pod="ha-pg-1"}'
values: '0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
alert_rule_test:
- eval_time: 39m
alertname: CNPGBackupFailed
exp_alerts: []
kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py
+563
View File
@@ -0,0 +1,563 @@
#!/usr/bin/env python3
"""Regenerate cnpg-backup_test.yaml from the REN-129 measurements.
The committed YAML is what CI runs; this script exists so the numbers in it are
auditable and reproducible from the measurement table rather than being a
hand-typed artifact nobody can re-derive. Run it from the repo root:
python3 kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py \
> kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml
Nothing in CI regenerates and diffs this — the promtool image has no Python, and
the defence against a hand-weakened suite is the mutation phase of
scripts/ci/promtool-rules.sh, not byte-equality with a generator. See README.md.
THE TRACE MODEL. Iris measured seven consecutive nights of
`cnpg_collector_pg_wal_archive_status{value="ready",pod="ha-pg-1"}` at 1-minute
steps and published, per night, a (peak, peak minute, fill seg/min, drain
seg/min) tuple. Those four numbers are consistent with one shape, which is the
shape every night has: flat 0, a linear ramp at `fill` per minute to `peak`,
then a linear decay at `drain` per minute back to 0. So a night is rebuilt as
ramp length = round(peak / fill) minutes
ramp start = peak minute - ramp length (value 0 at that minute)
during ramp = round(fill * minutes since ramp start)
at peak = peak (exact, not rounded from fill)
during decay = max(0, round(peak - drain * minutes since peak))
FIDELITY. The reconstruction reproduces Iris's published `Duration >5` column
exactly on 2026-09-20, -21 and -22, and runs one minute long on -23 and -24 and
two minutes long on -25 and -26. Every discrepancy is in the same direction:
the rebuild is never SHORTER above the threshold than the measurement. A linear
decay at the published mean drain rate holds the tail above 5 for slightly
longer than the real trace did, so each night is a marginally harder silence
test than the night it stands for. That is the safe direction for this suite,
whose whole job is to catch a rule that fires on a benign night. `--report`
prints the comparison; do not "fix" it toward the published column without
checking which way each night moves.
These are reconstructions, not the raw samples. They are faithful in the only
dimensions the rules read: how high the backlog goes, and how many consecutive
minutes it spends above each threshold. Do not read them as a recording.
"""
import argparse
import sys
# Iris's 7-night measurement, REN-129. `dur_gt5` is her published "Duration >5"
# column and is an INPUT TO THE SELF-CHECK ONLY — the traces are built from
# peak/fill/drain/peak_min, and dur_gt5 is what the result is compared against.
NIGHTS = [
# date peak peak_min fill drain dur_gt5
("2026-09-20", 106, "02:15", 35.3, 13.2, 10),
("2026-09-21", 100, "02:15", 33.3, 14.3, 9),
("2026-09-22", 101, "02:15", 33.7, 14.4, 9),
("2026-09-23", 119, "02:15", 39.7, 14.9, 9),
("2026-09-24", 56, "02:14", 18.7, 14.0, 5),
("2026-09-25", 91, "02:15", 22.8, 13.0, 8),
("2026-09-26", 113, "02:15", 28.5, 14.1, 9),
]
# The adversarial envelope: worst fill, worst peak, slowest drain observed in the
# week, combined into a night that did not happen but is inside what the cluster
# has demonstrated it can do. Nothing here is extrapolated past the measurements.
ENVELOPE = ("envelope", 119, "02:15", 39.7, 13.0, None)
# Series labels. `container`, `role` and `instance` are on the real series and
# are carried here on purpose: they are what `max by (namespace, pod)` has to
# aggregate away, and a test whose input is already aggregated cannot see whether
# the rule does that. `value="done"` is the cumulative counter that sits on the
# same metric name — it is in the input so that dropping the `value="ready"`
# selector from the rules is a detectable mutation and not a silent no-op.
NS = "home-assistant"
POD = "ha-pg-1"
def labels(pod=POD, value="ready", role="primary", ns=NS):
return (
"cnpg_collector_pg_wal_archive_status{"
f'namespace="{ns}",pod="{pod}",container="postgres",'
f'role="{role}",value="{value}"'
"}"
)
# Test window: t=0 is 02:08Z, 40 one-minute samples, so every night's burst sits
# inside it with lead-in and tail. promtool eval_times are relative to t=0, so
# every assertion below carries its UTC clock time in a comment.
#
# This MUST stay aligned with the peak minutes in NIGHTS. An earlier revision
# had `8 * 60 + 8` here (08:08Z), which put the whole window seven hours after
# every burst: all seven nights rebuilt as 40 zeroes and their ~90 silence
# assertions passed without exercising a single threshold. The
# assert_window_covers_bursts() call in main() exists so that cannot recur
# silently.
T0 = 2 * 60 + 8 # 02:08Z in minutes since midnight
SAMPLES = 40
def hhmm(minute):
return f"{minute // 60:02d}:{minute % 60:02d}"
def parse_hhmm(s):
h, m = s.split(":")
return int(h) * 60 + int(m)
def trace(peak, peak_min, fill, drain, n=SAMPLES, t0=T0):
"""Rebuild one night. See THE TRACE MODEL in the module docstring."""
ramp_len = max(1, round(peak / fill))
ramp_start = peak_min - ramp_len
out = []
for m in range(t0, t0 + n):
if m <= ramp_start:
v = 0
elif m < peak_min:
v = round(fill * (m - ramp_start))
elif m == peak_min:
v = peak
else:
v = max(0, round(peak - drain * (m - peak_min)))
out.append(v)
return out
def longest_run_above(vals, thr):
"""(length, index of last sample) of the longest run of vals > thr."""
best = (0, None)
run = 0
for i, v in enumerate(vals):
if v > thr:
run += 1
if run > best[0]:
best = (run, i)
else:
run = 0
return best
def pending_minutes(vals, thr):
"""Max minutes a `for` timer accumulates on `> thr` over this trace.
This is NOT the run length, and the difference is the whole margin
argument. Prometheus makes an alert pending at the first evaluation where
the expression is true and fires it when `now - activeAt >= for`, so a run
of N consecutive 1-minute samples above the threshold accumulates N-1
minutes of pending time, not N. A trace that spends 5 samples above 64
therefore does NOT fire `for: 5m`; it fires `for: 4m`. Quote this number,
not the run length, when arguing about a `for` window.
"""
n, _ = longest_run_above(vals, thr)
return max(0, n - 1)
def assert_window_covers_bursts():
"""Refuse to emit a suite whose trace window misses the bursts.
Guards the T0 bug described above T0: if the window and the peak minutes
ever drift apart again, every measured night rebuilds as flat zero and the
suite passes while testing nothing. A night that is flat 0 cannot be a
silence test for a burst, so treat it as a generator fault, not a result.
"""
for date, peak, pm, fill, drain, _dur in NIGHTS + [ENVELOPE]:
vals = trace(peak, parse_hhmm(pm), fill, drain)
if max(vals) != peak:
raise SystemExit(
f"generator fault: {date} rebuilt with max {max(vals)}, expected "
f"peak {peak}. The T0={T0} window ({hhmm(T0)}Z.."
f"{hhmm(T0 + SAMPLES - 1)}Z) does not cover peak minute {pm}Z. "
"Fix T0 or SAMPLES; do not emit a suite of flat-zero nights."
)
def counter(start, step, n=SAMPLES):
return [start + step * i for i in range(n)]
def fmt(vals):
return " ".join(str(v) for v in vals)
def eval_minutes(vals):
"""The minutes where a `for` window is closest to elapsing, so the tightest
places to assert silence. For each threshold: the last sample of the longest
run above it (the moment maximum pending time has accumulated). Plus the
peak, plus two minutes after the backlog has settled back to 0."""
picks = set()
for thr in (5, 64):
_, last = longest_run_above(vals, thr)
if last is not None:
picks.add(last)
picks.add(max(range(len(vals)), key=lambda i: vals[i]))
settled = next(
(i for i in range(len(vals) - 1, -1, -1) if vals[i] > 0),
len(vals) - 1,
)
picks.add(min(settled + 2, len(vals) - 1))
return sorted(picks)
L = []
def w(line=""):
L.append(line)
def burst_case(name, date, peak, peak_min_s, fill, drain, why):
peak_min = parse_hhmm(peak_min_s)
vals = trace(peak, peak_min, fill, drain)
n5, _ = longest_run_above(vals, 5)
n64, _ = longest_run_above(vals, 64)
w(f" # {why}")
w(
f" # peak {peak} at {peak_min_s}Z, fill {fill}/min, drain {drain}/min"
f" -> {n5} samples above 5 ({pending_minutes(vals, 5)}m pending vs"
f" for: 15m), {n64} above 64 ({pending_minutes(vals, 64)}m pending vs"
f" for: 8m)."
)
w(f" - name: {name}")
w(" interval: 1m")
w(" input_series:")
w(f" - series: '{labels()}'")
w(f" # {hhmm(T0)}Z .. {hhmm(T0 + SAMPLES - 1)}Z, one sample per minute")
w(f" values: '{fmt(vals)}'")
w(f" - series: '{labels(value='done')}'")
w(" # The cumulative archived counter on the same metric name. The")
w(" # rules must never read it; see mutations.txt:no-value-selector.")
w(f" values: '{fmt(counter(148_000, 14))}'")
w(" alert_rule_test:")
for i in eval_minutes(vals):
clock = hhmm(T0 + i)
for alert in ("CNPGWalArchiveBacklog", "CNPGWalArchiveBacklogSurge"):
w(f" # {clock}Z, backlog {vals[i]}")
w(f" - eval_time: {i}m")
w(f" alertname: {alert}")
w(" exp_alerts: []")
w("")
return vals, n5, n64
def main():
ap = argparse.ArgumentParser()
ap.add_argument(
"--report",
action="store_true",
help="print the reconstruction self-check table instead of the suite",
)
args = ap.parse_args()
assert_window_covers_bursts()
if args.report:
# `pend` columns are what the `for` windows are actually compared
# against; see pending_minutes(). `>5 pub` is Iris's published column
# and is the only number here that is not derived by this script.
print(f"{'night':12} {'peak':>5} {'fill':>6} {'drain':>6} "
f"{'>5 pub':>7} {'>5 n':>6} {'>5 pend':>8} "
f"{'>64 n':>6} {'>64 pend':>9}")
for date, peak, pm, fill, drain, dur in NIGHTS + [ENVELOPE]:
vals = trace(peak, parse_hhmm(pm), fill, drain)
n5, _ = longest_run_above(vals, 5)
n64, _ = longest_run_above(vals, 64)
print(f"{date:12} {peak:5} {fill:6} {drain:6} "
f"{('-' if dur is None else dur):>7} {n5:>6} "
f"{pending_minutes(vals, 5):>8} "
f"{n64:>6} {pending_minutes(vals, 64):>9}")
print()
print("for: 15m on >5 must exceed every '>5 pend' above.")
print("for: 8m on >64 must exceed every '>64 pend' above.")
return 0
w("# GENERATED by generate-cnpg-backup-test.py — see README.md in this")
w("# directory before editing by hand.")
w("#")
w("# Unit tests for kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml.")
w("# Every assertion below is either one of Iris's seven measured nights")
w("# (REN-129) rebuilt from its published (peak, fill, drain) numbers, an")
w("# adversarial combination of the extremes of those numbers, or a failure")
w("# this cluster has not had but the rules exist to catch.")
w("#")
w("# `rule_files` entries are resolved RELATIVE TO THIS FILE'S DIRECTORY, not")
w("# to the working directory promtool was invoked from — promtool joins")
w("# dirname(testfile) onto each entry. So this is `../`, and it resolves in")
w("# the mirrored tree scripts/ci/promtool-rules.sh builds, where the")
w("# `spec:`-stripped manifests sit in the same layout as the real ones.")
w("#")
w("# A path that does NOT resolve is only a WARNING to promtool: it loads zero")
w("# rules, every exp_alerts: [] assertion passes against nothing, and the")
w("# suite exits 0. promtool-rules.sh resolves every entry itself before")
w("# running and fails on a miss; that check is the only thing standing")
w("# between a typo here and a green pipeline that tests nothing.")
w("rule_files:")
w(" - ../prometheusrule-cnpg-backup.yaml")
w("evaluation_interval: 1m")
w("")
w("tests:")
w(" # ---------------------------------------------------------------------")
w(" # 1-7. THE SEVEN MEASURED NIGHTS. Both rules must stay silent through")
w(" # every one of them. This is the half of the suite that has already")
w(" # caught something: the surge rule shipped for review at `for: 2m` and")
w(" # fires here on 6 of these 7 nights (mutations.txt:surge-for-2m).")
w(" # ---------------------------------------------------------------------")
checks = []
for date, peak, pm, fill, drain, dur in NIGHTS:
vals, n5, n64 = burst_case(
f"nightly burst {date} — recorder purge, benign, self-clearing",
date, peak, pm, fill, drain,
f"{date}: the nightly Home Assistant recorder purge at 04:12 local.",
)
checks.append((date, dur, n5))
w(" # ---------------------------------------------------------------------")
w(" # 8. THE ADVERSARIAL ENVELOPE. A night that did not happen, built from")
w(" # the worst fill (39.7/min, 09-23), the highest peak (119, 09-23) and")
w(" # the slowest drain (13.0/min, 09-25) in the week. Inside what the")
w(" # cluster has demonstrated, worse than anything it actually did.")
w(" #")
w(" # This is where the two `for` windows get measured for margin rather")
w(" # than asserted about. Margin is PENDING time, which is one less than")
w(" # the run length — an alert goes pending on the first true evaluation")
w(" # and fires at activeAt + for, so 6 consecutive samples above 64")
w(" # accumulate 5 minutes, not 6:")
w(" #")
w(" # above 5: 11 samples -> 10m pending vs `for: 15m` -> 5m margin")
w(" # above 64: 6 samples -> 5m pending vs `for: 8m` -> 3m margin")
w(" #")
w(" # The worst MEASURED night accumulates 4m above 64 (5 samples, 09-20")
w(" # and 09-23), so the envelope is one minute tighter than anything that")
w(" # actually happened. 3 minutes is the number to defend if anyone")
w(" # proposes lowering `for: 8m` — and mutations.txt:surge-for-5m is the")
w(" # control that makes this case load-bearing rather than decorative: at")
w(" # `for: 5m` the surge rule is still silent on all seven real nights and")
w(" # fires only here. Delete this case and that mutation stops being")
w(" # caught.")
w(" # ---------------------------------------------------------------------")
burst_case(
"adversarial envelope — worst measured fill and peak, slowest drain",
*ENVELOPE[:5],
why="Synthetic: max fill 39.7 + max peak 119 + min drain 13.0.",
)
# ---- 9. stalled archiver, above the surge threshold -------------------
stalled = [0] * 5 + [70] * 35
w(" # ---------------------------------------------------------------------")
w(" # 9. THE FAILURE BOTH RULES EXIST FOR: an archiver that stopped with")
w(" # 70 segments (1.1 GiB) queued. Not a burst — it does not drain.")
w(" # Surge fires first (8m after the backlog crosses 64), the")
w(" # load-bearing rule 7 minutes later (15m after it crosses 5). Both")
w(" # assertions bracket the transition: one minute early nothing is")
w(" # firing, on the minute it is.")
w(" # ---------------------------------------------------------------------")
w(" - name: stalled archiver at 70 segments — both rules must fire")
w(" interval: 1m")
w(" input_series:")
w(f" - series: '{labels()}'")
w(f" values: '{fmt(stalled)}'")
w(f" - series: '{labels(value='done')}'")
w(" # Flat, because a stalled archiver archives nothing. The rules do")
w(" # not read this; it is here so the input stays realistic.")
w(f" values: '{fmt([148_000] * SAMPLES)}'")
w(" alert_rule_test:")
w(" # Backlog crosses 64 at t=5m, so `for: 8m` elapses at t=13m.")
w(" - eval_time: 12m")
w(" alertname: CNPGWalArchiveBacklogSurge")
w(" exp_alerts: []")
w(" - eval_time: 13m")
w(" alertname: CNPGWalArchiveBacklogSurge")
w(" exp_alerts:")
w(" # No `role` label: proof that `max by (namespace, pod)`")
w(" # aggregated it away. See mutations.txt:no-max-by.")
w(" - exp_labels:")
w(" severity: warning")
w(f" namespace: {NS}")
w(f" pod: {POD}")
w(" exp_annotations:")
w(f' summary: "CNPG WAL archiving cannot keep up on {NS}/{POD}"')
w(" description: \"70 WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129).\"")
w(" # Backlog crosses 5 at t=5m, so `for: 15m` elapses at t=20m.")
w(" - eval_time: 19m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts: []")
w(" - eval_time: 20m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts:")
w(" - exp_labels:")
w(" severity: warning")
w(f" namespace: {NS}")
w(f" pod: {POD}")
w(" exp_annotations:")
w(f' summary: "CNPG WAL archiving is falling behind on {NS}/{POD}"')
w(" description: \"70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
w("")
# ---- 10. low stall, below the surge threshold -------------------------
low = [0] * 5 + [12] * 35
w(" # ---------------------------------------------------------------------")
w(" # 10. THE STALL THE SURGE RULE CANNOT SEE: 12 segments, stuck. Under")
w(" # 1 GiB forever, so the surge rule is right to stay silent — and this")
w(" # is exactly the shape that makes the >5 rule load-bearing rather than")
w(" # a slower copy of it. Point-in-time recovery is just as broken at 12")
w(" # segments as at 70.")
w(" # ---------------------------------------------------------------------")
w(" - name: low stall at 12 segments — only the >5 rule may fire")
w(" interval: 1m")
w(" input_series:")
w(f" - series: '{labels()}'")
w(f" values: '{fmt(low)}'")
w(" alert_rule_test:")
w(" - eval_time: 20m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts:")
w(" - exp_labels:")
w(" severity: warning")
w(f" namespace: {NS}")
w(f" pod: {POD}")
w(" exp_annotations:")
w(f' summary: "CNPG WAL archiving is falling behind on {NS}/{POD}"')
w(" description: \"12 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
w(" # 34 minutes at 12 segments and the surge rule is still quiet.")
w(" - eval_time: 39m")
w(" alertname: CNPGWalArchiveBacklogSurge")
w(" exp_alerts: []")
w("")
# ---- 11. exactly at the threshold ------------------------------------
w(" # ---------------------------------------------------------------------")
w(" # 11. PINNED AT EXACTLY 5 for the whole window. `> 5` is strict, so")
w(" # this is silence. A rule written `>= 5` would page on it, and a rule")
w(" # that had lost its `value=\"ready\"` selector would page on the `done`")
w(" # counter regardless of what `ready` does — which is what the second")
w(" # series here is for.")
w(" # ---------------------------------------------------------------------")
w(" - name: pinned at exactly 5 — strict > means silence")
w(" interval: 1m")
w(" input_series:")
w(f" - series: '{labels()}'")
w(f" values: '{fmt([5] * SAMPLES)}'")
w(f" - series: '{labels(value='done')}'")
w(f" values: '{fmt(counter(148_000, 14))}'")
w(" alert_rule_test:")
w(" - eval_time: 39m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts: []")
w(" - eval_time: 39m")
w(" alertname: CNPGWalArchiveBacklogSurge")
w(" exp_alerts: []")
w("")
# ---- 12. role flip ---------------------------------------------------
w(" # ---------------------------------------------------------------------")
w(" # 12. FAILOVER DURING A STALL — the case `max by (namespace, pod)` was")
w(" # added for. The backlog starts at t=5m on the primary; at t=11m the")
w(" # instance is promoted and `role` changes, which is a NEW SERIES to")
w(" # Prometheus and restarts the `for` timer on the raw expression.")
w(" # Aggregating `role` away keeps one continuous series, so the page")
w(" # still lands 15m after the backlog started (t=20m) rather than 15m")
w(" # after the failover. Without `max by` this case is silent at t=20m —")
w(" # see mutations.txt:no-max-by.")
w(" # ---------------------------------------------------------------------")
w(" - name: role flip mid-stall — for timer must not restart")
w(" interval: 1m")
w(" input_series:")
w(f" - series: '{labels(role='primary')}'")
w(" # t=0..10m, then this series ends.")
w(f" values: '{fmt([0] * 5 + [70] * 6)}'")
w(f" - series: '{labels(role='replica')}'")
w(" # Starts at t=11m. `_` is a gap, not a zero.")
w(f" values: '{'_ ' * 11}{fmt([70] * 29)}'")
w(" alert_rule_test:")
w(" - eval_time: 19m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts: []")
w(" - eval_time: 20m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts:")
w(" - exp_labels:")
w(" severity: warning")
w(f" namespace: {NS}")
w(f" pod: {POD}")
w(" exp_annotations:")
w(f' summary: "CNPG WAL archiving is falling behind on {NS}/{POD}"')
w(" description: \"70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
w("")
# ---- 13. one stalled pod among healthy ones --------------------------
w(" # ---------------------------------------------------------------------")
w(" # 13. ONE STALLED INSTANCE AMONG HEALTHY ONES. There are 12 CNPG")
w(" # instances in this cluster. `by (namespace, pod)` has to keep them")
w(" # apart: exactly one alert, naming the pod that is actually stuck, so")
w(" # the 04:00 reader is not sent to look at the wrong Postgres.")
w(" # ---------------------------------------------------------------------")
w(" - name: one stalled pod among healthy ones — exactly one alert")
w(" interval: 1m")
w(" input_series:")
w(f" - series: '{labels(pod='ha-pg-1')}'")
w(f" values: '{fmt([0] * 5 + [70] * 35)}'")
w(f" - series: '{labels(pod='ha-pg-2', role='replica')}'")
w(f" values: '{fmt([0] * SAMPLES)}'")
w(f" - series: '{labels(pod='immich-pg-1', ns='immich')}'")
w(f" values: '{fmt([0] * SAMPLES)}'")
w(" alert_rule_test:")
w(" - eval_time: 20m")
w(" alertname: CNPGWalArchiveBacklog")
w(" exp_alerts:")
w(" - exp_labels:")
w(" severity: warning")
w(f" namespace: {NS}")
w(" pod: ha-pg-1")
w(" exp_annotations:")
w(f' summary: "CNPG WAL archiving is falling behind on {NS}/ha-pg-1"')
w(" description: \"70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
w("")
# ---- 14. CNPGBackupFailed --------------------------------------------
w(" # ---------------------------------------------------------------------")
w(" # 14. THE EPOCH-ZERO GUARD on CNPGBackupFailed. The metric is a unix")
w(" # timestamp and reads 0 when no failure was ever recorded, so the rule")
w(" # carries a `> 0` guard to stop that default being read as \"failed in")
w(" # 1970\". Held at 0 for the whole window: silence. Nothing else in this")
w(" # suite exercises that rule, and a dropped guard would page on every")
w(" # instance in the cluster forever.")
w(" # ---------------------------------------------------------------------")
w(" - name: never-failed backup reads timestamp 0 — must not fire")
w(" interval: 1m")
w(" input_series:")
w(f" - series: 'cnpg_collector_last_failed_backup_timestamp{{namespace=\"{NS}\",pod=\"{POD}\"}}'")
w(f" values: '{fmt([0] * SAMPLES)}'")
w(" alert_rule_test:")
w(" - eval_time: 39m")
w(" alertname: CNPGBackupFailed")
w(" exp_alerts: []")
print("\n".join(L))
# Fidelity self-check on stderr so it is visible when regenerating but does
# not contaminate the suite on stdout. Longer than measured is tolerated and
# expected (see FIDELITY in the module docstring); SHORTER is not, because a
# night that clears the threshold sooner than the real one is a weaker
# silence test than the night it claims to stand for.
short = [(d, m, r) for d, m, r in checks if r < m]
if short:
print(
"FAIL: reconstruction is SHORTER above 5 than the measurement on "
+ ", ".join(f"{d} measured {m} rebuilt {r}" for d, m, r in short)
+ " — these nights are weaker silence tests than the real ones. "
"Refusing to vouch for the suite.",
file=sys.stderr,
)
return 1
long_ = [(d, m, r) for d, m, r in checks if r > m]
if long_:
print(
f"NOTE: {len(long_)} of {len(checks)} nights rebuild LONGER above 5 "
"than the published column (conservative direction): "
+ ", ".join(f"{d} {m}->{r}" for d, m, r in long_),
file=sys.stderr,
)
return 0
if __name__ == "__main__":
sys.exit(main())
kubernetes/apps/monitoring-rules/tests/mutations.txt
+72
View File
@@ -0,0 +1,72 @@
# Negative controls for the promtool suites in this directory.
#
# A unit-test suite that passes proves the rules behave as asserted. It does not
# prove the assertions are worth anything: a suite whose `rule_files` path does
# not resolve, or whose traces are all zero, passes just as green. Both of those
# happened while this suite was being written, and neither was visible from the
# exit code.
#
# So scripts/ci/promtool-rules.sh runs a second phase. For each mutation below
# it damages ONE value the rules depend on, re-runs every suite, and requires the
# run to FAIL. A mutation the suite does not notice is a hole in the suite, and
# CI reports it as such.
#
# FORMAT — three `|`-separated fields, no `|` inside any of them:
#
# name | manifest path relative to the repo root | sed program
#
# The sed program is applied to the `spec:`-stripped copy in the mirrored tree,
# so its indentation is two spaces shallower than the manifest on disk. Anchor
# on `^[[:space:]]*` rather than a fixed indent. `#` in the first column is a
# comment; blank lines are ignored.
#
# The script also requires each sed program to CHANGE the file. A mutation that
# matches nothing is reported as a failure, not skipped — otherwise renaming a
# rule or rewording an expression silently turns its control into a no-op and
# the suite goes back to vouching for itself.
#
# `detected by` notes below are for the reader. The machine check is only "the
# suite must fail"; nothing parses them.
# The defect this whole exercise exists because of. The surge rule was written
# `for: 2m` and reviewed at that value. Six of Iris's seven measured nights
# accumulate 3-4 minutes above 64, so it fires on all six — two owner emails a
# night, 04:16 and 04:22 local, for a backlog that clears itself.
# Detected by: cases 1-7, the measured nights.
surge-for-2m|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|s/^\([[:space:]]*\)for: 8m$/\1for: 2m/
# The margin control, and the only reason case 8 is in the suite. At `for: 5m`
# the surge rule is still silent on all seven REAL nights (worst is 4m pending)
# and fires only on the adversarial envelope (5m pending). Without the envelope
# case, `for: 8m` could be cut to 5m with the suite staying green.
# Detected by: case 8, the adversarial envelope.
surge-for-5m|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|s/^\([[:space:]]*\)for: 8m$/\1for: 5m/
# The value REN-129 changed, in the direction it changed it from. At 30m the
# load-bearing rule does not page until 35 minutes into a stall that started at
# t=5m, so the assertion at t=20m sees nothing. This is the control that stops
# the 15m being quietly walked back.
# Detected by: cases 9, 10, 12, 13 — every case that expects a page.
backlog-for-30m|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|s/^\([[:space:]]*\)for: 15m$/\1for: 30m/
# Drop the aggregation, keeping the selector. `role` flips on failover; a
# changed label is a new series to Prometheus and restarts the `for` timer, so
# a backlog that persists across a promotion pages 15m after the promotion
# instead of 15m after the backlog. Scoped to `expr:` lines because
# `max by (namespace, pod)` also appears in this manifest's comments, where
# rewriting it would mutate the documentation and nothing else.
# Detected by: case 12, the role flip.
no-max-by|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|/^[[:space:]]*expr:/s/max by (namespace, pod) (\(cnpg_collector_pg_wal_archive_status{value="ready"}\))/\1/
# Drop the `value="ready"` selector. The same metric name also carries
# `value="done"`, a monotonic counter in the hundreds of thousands, so both
# rules would fire permanently on every CNPG instance in the cluster. The
# measured-night cases carry that counter series in their input specifically so
# this is detectable rather than a silent no-op.
# Detected by: cases 1-8 and 11 — every case with a `done` series.
no-value-selector|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|/^[[:space:]]*expr:/s/{value="ready"}//
# Loosen the comparison on the load-bearing rule. `> 5` is strict; `>= 5` pages
# on a backlog sitting at exactly five segments, which is 80 MiB and normal.
# Detected by: case 11, pinned at exactly 5.
backlog-ge-5|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|/^[[:space:]]*expr:/s/> 5$/>= 5/
scripts/ci/promtool-rules.sh
Executable
+306
View File
@@ -0,0 +1,306 @@
#!/bin/sh
# Machine-check the PrometheusRule manifests in this repo: parse every rule with
# `promtool check rules`, run the committed unit suites with `promtool test
# rules`, then damage the rules on purpose and require the suites to notice.
#
# Run: sh scripts/ci/promtool-rules.sh [repo-root]
# Needs: promtool on PATH (3.x — see the version note below), sh, find, sed, awk.
# Offline: no cluster, no registry, no secrets, no network.
# Tests: scripts/ci/test-promtool-rules.sh (runs the guards against fixture
# trees with a stub promtool; no real promtool needed).
#
# WHY THE MIRROR. promtool reads Prometheus rule files, which are bare `groups:`
# documents. A PrometheusRule is a Kubernetes object carrying the same thing
# under `spec:`, so promtool cannot read one directly. This script copies every
# PrometheusRule into a temporary tree at the SAME relative path with `spec:`
# stripped and the body dedented, and runs promtool there. Nothing is written
# back into the checkout.
#
# WHY THE MUTATION PHASE. Two failure modes bit while the first suite was being
# written, and neither changed promtool's exit code:
#
# * a `rule_files:` entry that did not resolve. promtool prints
# "WARNING: no file match pattern ...", loads ZERO rules, and then every
# `exp_alerts: []` assertion passes against an empty rule set. Ninety
# silence assertions, green, testing nothing.
# * traces generated against the wrong clock window, so every input series was
# flat zero. Same outcome: green, testing nothing.
#
# A suite that passes tells you the rules match the assertions. It does not tell
# you the assertions reach the rules. The two guards in step 4 and step 5 cover
# the first failure mode by name; the mutation phase in step 6 is the backstop
# for the general case, and is the only part of this that would have caught the
# second.
#
# PROMTOOL VERSION. Must match the Prometheus the cluster actually runs, which is
# whatever kube-prometheus-stack >= 72.0.0 ships — 3.x. Alert `for` evaluation is
# version-sensitive, so a 2.x promtool would be checking a different evaluator
# than production and the margins asserted in the suites would be claims about
# the wrong program. The exact pin lives on the `image:` line in
# .woodpecker/ci-manifests-test.yml; this script asserts only the major version,
# so a patch bump there needs no change here.
set -eu
ROOT="${1:-.}"
cd "$ROOT"
# Floors against silent discovery failure. Both are LOOSE on purpose. A floor
# tight enough to catch the loss of one file sits inside normal churn, so it
# fails spuriously — and the fix a hurried person reaches for is lowering the
# floor, which is how a guard becomes decoration. These catch total failure and
# gross partial loss, which is the failure that actually happens: a `find`
# expression the image's BusyBox does not support, or a moved directory.
#
# Same reasoning, and the same asymmetry, as the floors in
# .woodpecker/ci-manifests-test.yml. 17 PrometheusRules and 1 suite exist today.
RULE_FLOOR="${PROMTOOL_RULE_FLOOR:-12}"
SUITE_FLOOR="${PROMTOOL_SUITE_FLOOR:-1}"
MUTATION_FLOOR="${PROMTOOL_MUTATION_FLOOR:-1}"
TESTS_DIR="${PROMTOOL_TESTS_DIR:-kubernetes/apps/monitoring-rules/tests}"
MUTATIONS="${PROMTOOL_MUTATIONS:-$TESTS_DIR/mutations.txt}"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT INT TERM
MIRROR="$WORK/mirror" # stripped manifests + suites, at repo-relative paths
MUTANT="$WORK/mutant" # a damaged copy of the above, one mutation at a time
mkdir -p "$MIRROR"
die() { echo "FAIL: $*" >&2; exit 1; }
command -v promtool >/dev/null 2>&1 || die "promtool is not on PATH."
# `promtool --version` writes to stderr on some builds, hence the 2>&1.
PROMTOOL_VERSION="$(promtool --version 2>&1 \
| sed -n 's/^promtool, version \([0-9][0-9.]*\).*/\1/p' | head -1)"
[ -n "$PROMTOOL_VERSION" ] || die "cannot parse 'promtool --version' output."
case "$PROMTOOL_VERSION" in
3.*) : ;;
*) die "promtool $PROMTOOL_VERSION is not 3.x. The cluster runs Prometheus 3
(kube-prometheus-stack >= 72.0.0) and alert 'for' evaluation is
version-sensitive, so a 2.x promtool would check a different evaluator
than production. Fix the image in .woodpecker/ci-manifests-test.yml." ;;
esac
echo "promtool $PROMTOOL_VERSION"
# ---------------------------------------------------------------------------
# 1. Discover PrometheusRule manifests.
# ---------------------------------------------------------------------------
# NOT `find -printf` and NOT `grep -r --include`: this runs on the BusyBox find
# and grep in prom/prometheus, which reject both. `find -print0 | xargs -0 grep`
# is the portable shape, and unlike a command substitution it does not run into
# ARG_MAX on a tree this wide. Anchoring `^kind:` at column 0 keeps the match to
# the top-level field and away from anything nested or quoted in a comment.
find kubernetes -name '*.yaml' -type f -print0 2>/dev/null \
| xargs -0 grep -l '^kind: PrometheusRule' 2>/dev/null \
| sort > "$WORK/rules" || true
rule_count=$(grep -c . "$WORK/rules" || true)
if [ "$rule_count" -lt "$RULE_FLOOR" ]; then
echo "FAIL: found only $rule_count PrometheusRule manifests; floor is $RULE_FLOOR." >&2
echo " If the tree legitimately shrank, lower PROMTOOL_RULE_FLOOR in" >&2
echo " .woodpecker/ci-manifests-test.yml — do not delete the guard." >&2
echo " Otherwise discovery is broken, and a check that found nothing" >&2
echo " must not report success." >&2
exit 1
fi
echo "discovered $rule_count PrometheusRule manifests"
# ---------------------------------------------------------------------------
# 2. Mirror each manifest as a bare rule file, and parse it.
# ---------------------------------------------------------------------------
# The awk strips the `spec:` line, dedents its block by two, and stops at the
# first line back at column 0 — so a trailing top-level key (`status:`) is
# dropped rather than emitted at the wrong depth. Blank lines pass through
# unchanged; they carry no indentation to dedent.
fail=0
while IFS= read -r f; do
[ -n "$f" ] || continue
mkdir -p "$MIRROR/$(dirname "$f")"
awk '
!ins && /^spec:[ \t]*$/ { ins = 1; next }
ins {
if ($0 == "") { print ""; next }
if (substr($0, 1, 2) == " ") { print substr($0, 3); next }
exit
}
' "$f" > "$MIRROR/$f"
# `promtool check rules` reports "SUCCESS: 0 rules found" on an empty document
# and exits 0, so a strip that produced nothing would pass. Require the result
# to actually be a rule file.
first="$(sed -n '/^[[:space:]]*$/d; p' "$MIRROR/$f" | head -1)"
if [ "$first" != "groups:" ]; then
echo "FAIL $f" >&2
echo " stripping spec: yielded [$first], expected [groups:]." >&2
echo " This manifest's rules are NOT being checked. Either spec: is" >&2
echo " not a top-level key here, or its body is not a groups: list." >&2
fail=1
continue
fi
if out=$(promtool check rules "$MIRROR/$f" 2>&1); then
echo "ok $f — $(echo "$out" | sed -n 's/.*\(SUCCESS: .*\)/\1/p' | head -1)"
else
echo "FAIL $f" >&2
echo "$out" | sed 's/^/ /' >&2
fail=1
fi
done < "$WORK/rules"
[ "$fail" -eq 0 ] || exit 1
# ---------------------------------------------------------------------------
# 3. Discover the committed unit suites and copy them into the mirror.
# ---------------------------------------------------------------------------
find "$TESTS_DIR" -name '*_test.yaml' -type f 2>/dev/null | sort > "$WORK/suites" || true
suite_count=$(grep -c . "$WORK/suites" || true)
if [ "$suite_count" -lt "$SUITE_FLOOR" ]; then
echo "FAIL: found only $suite_count *_test.yaml suites under $TESTS_DIR;" >&2
echo " floor is $SUITE_FLOOR. Discovery is broken, or the suites were" >&2
echo " deleted. Either way this step must not report success." >&2
exit 1
fi
echo "discovered $suite_count unit suite(s)"
while IFS= read -r t; do
[ -n "$t" ] || continue
mkdir -p "$MIRROR/$(dirname "$t")"
cp "$t" "$MIRROR/$t"
done < "$WORK/suites"
# ---------------------------------------------------------------------------
# 4. Resolve every rule_files entry BEFORE running anything.
# ---------------------------------------------------------------------------
# The guard for the first failure mode at the top of this file. promtool joins
# dirname(suite) onto each `rule_files` entry and, when the result matches
# nothing, prints a WARNING and carries on with zero rules loaded — so the suite
# passes having asserted nothing. There is no promtool flag that makes it fatal,
# so resolve the paths here and refuse.
#
# The extraction is deliberately narrow: `rule_files:` at column 0, then the
# ` - path` lines under it, stopping at the next column-0 key. The flow form
# (`rule_files: [a, b]`) is not parsed — so it is rejected by name below rather
# than skipped in silence, which would reopen the hole this closes.
while IFS= read -r t; do
[ -n "$t" ] || continue
if grep -q '^rule_files:[[:space:]]*[^[:space:]]' "$t"; then
die "$t writes rule_files inline. Use the block form (rule_files: followed
by ' - path' lines): this script does not resolve the inline form, and
an unresolved path there would be checked by nobody."
fi
awk '
/^rule_files:[[:space:]]*$/ { inrf = 1; next }
inrf && /^[[:space:]]*-[[:space:]]*/ {
sub(/^[[:space:]]*-[[:space:]]*/, "")
sub(/[[:space:]]*(#.*)?$/, "")
gsub(/^["'"'"']|["'"'"']$/, "")
print
next
}
inrf && /^[^[:space:]]/ { inrf = 0 }
' "$t" > "$WORK/rf"
n=$(grep -c . "$WORK/rf" || true)
[ "$n" -gt 0 ] || die "$t declares no rule_files. It would load zero rules and
pass every exp_alerts: [] assertion against nothing."
while IFS= read -r rf; do
[ -n "$rf" ] || continue
case "$rf" in
/*) resolved="$rf" ;;
*) resolved="$MIRROR/$(dirname "$t")/$rf" ;;
esac
[ -f "$resolved" ] || die "$t references rule_files entry [$rf], which does
not resolve. promtool would treat that as a WARNING, load zero rules and
pass. Entries are relative to the SUITE's own directory, not the repo
root: from $TESTS_DIR, a manifest in the parent is '../name.yaml'."
done < "$WORK/rf"
echo "ok $t — $n rule_files entry/entries resolved"
done < "$WORK/suites"
# ---------------------------------------------------------------------------
# 5. Run the suites.
# ---------------------------------------------------------------------------
# run_suites <dir> -> 0 if every suite passed, 1 otherwise. Output goes to
# $WORK/out so the mutation phase can stay quiet when a mutation is correctly
# caught, and the caller can print it when the result is not what it expected.
run_suites() {
_dir="$1"
_rc=0
: > "$WORK/out"
while IFS= read -r _t; do
[ -n "$_t" ] || continue
if ! (cd "$_dir" && promtool test rules "$_t") >> "$WORK/out" 2>&1; then
_rc=1
fi
done < "$WORK/suites"
# Belt and braces on the same trap step 4 covers by path: if a future promtool
# reaches a missing rule file by a route this script does not model, the
# WARNING is still the tell. A warning here is a failure.
if grep -q 'WARNING: no file match pattern' "$WORK/out"; then
echo " (promtool warned about an unresolved rule file — zero rules loaded)" \
>> "$WORK/out"
_rc=1
fi
return $_rc
}
if run_suites "$MIRROR"; then
sed 's/^/ /' "$WORK/out"
echo "ok all $suite_count suite(s) pass against the rules as committed"
else
echo "FAIL: the committed rules do not satisfy their own unit suites." >&2
sed 's/^/ /' "$WORK/out" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# 6. Mutation phase: break the rules on purpose; the suites must notice.
# ---------------------------------------------------------------------------
[ -f "$MUTATIONS" ] || die "$MUTATIONS is missing. Without it the suites vouch
for themselves — see WHY THE MUTATION PHASE at the top of this script."
mutation_count=0
while IFS='|' read -r name target program; do
case "${name:-}" in ''|\#*) continue ;; esac
[ -n "${target:-}" ] && [ -n "${program:-}" ] \
|| die "$MUTATIONS: malformed entry [$name]; expected name|path|sed-program."
[ -f "$MIRROR/$target" ] \
|| die "$MUTATIONS: mutation [$name] targets $target, which is not one of
the $rule_count discovered PrometheusRule manifests."
rm -rf "$MUTANT"
cp -R "$MIRROR" "$MUTANT"
sed "$program" "$MIRROR/$target" > "$MUTANT/$target"
# A sed program that matches nothing leaves the tree undamaged, the suites
# pass, and the mutation reads as undetected — so it would already fail below.
# It is checked separately because the two cases need different repairs: "the
# suite got weaker" means fix the suite, "the rule was renamed or reworded"
# means fix the sed program.
if cmp -s "$MIRROR/$target" "$MUTANT/$target"; then
die "mutation [$name] changed nothing in $target. Its sed program no longer
matches — the rule was probably renamed or reworded. Update the program
in $MUTATIONS; do not delete the control."
fi
if run_suites "$MUTANT"; then
echo "FAIL: mutation [$name] was NOT detected — the suites still pass with" >&2
echo " $target damaged. Whatever this mutation breaks is uncovered." >&2
echo " Diff applied to the stripped manifest:" >&2
diff "$MIRROR/$target" "$MUTANT/$target" | sed 's/^/ /' >&2 || true
exit 1
fi
echo "ok mutation [$name] detected"
mutation_count=$((mutation_count + 1))
done < "$MUTATIONS"
if [ "$mutation_count" -lt "$MUTATION_FLOOR" ]; then
echo "FAIL: ran only $mutation_count mutations; floor is $MUTATION_FLOOR." >&2
echo " An empty mutation phase leaves the suites vouching for" >&2
echo " themselves, which is the thing it exists to prevent." >&2
exit 1
fi
echo "PASS $rule_count manifests parsed, $suite_count suite(s) green," \
"$mutation_count mutation(s) caught"
scripts/ci/test-promtool-rules.sh
Executable
+264
View File
@@ -0,0 +1,264 @@
#!/bin/sh
# Offline self-test for the guards in promtool-rules.sh.
#
# Run: sh scripts/ci/test-promtool-rules.sh
#
# No real promtool, no network, no cluster — a stub `promtool` on PATH plus small
# fixture trees. That is deliberate: what is being tested here is not whether
# promtool evaluates PromQL correctly (it does, and ci-manifests-test.yml checks
# the real rules with the real binary) but whether promtool-rules.sh REFUSES TO
# REPORT SUCCESS when it has actually checked nothing. Those are the cases a
# green pipeline cannot distinguish from real coverage, so they need their own
# test, and stubbing promtool is what lets this run on the ci-claude image next
# to the other script tests.
#
# Every case below is a failure mode that occurred or was one edit away:
#
# empty-discovery the floor guard the issue asked for
# no-suites same, for the test files
# unresolved-rule-file the one that actually shipped — promtool WARNs, loads
# zero rules, and passes every silence assertion
# warning-only the same hole reached by a route the path check cannot
# see; caught by scanning promtool's output
# inline-rule-files a shape the resolver cannot parse, rejected by name
# rather than skipped in silence
# empty-rule-files a suite that declares no rules at all
# unstrippable-spec a manifest the spec: strip does not turn into a rule
# file, which promtool would report as "0 rules found"
# mutation-no-op a control whose sed no longer matches anything
# mutation-undetected a control the suite does not catch
# happy-path all of the above satisfied: must PASS
set -eu
HERE="$(cd "$(dirname "$0")" && pwd)"
SUT="$HERE/promtool-rules.sh"
[ -f "$SUT" ] || { echo "FAIL: $SUT not found"; exit 1; }
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT INT TERM
fails=0
# --- the stub -------------------------------------------------------------
# `check rules` always succeeds. `test rules` fails if the tree it is run from
# contains the sentinel a mutation introduces, which is what makes the mutation
# loop genuinely exercised rather than mocked: the baseline tree has no
# sentinel and passes, a mutated tree has one and fails.
#
# STUB_VERSION what to report for --version
# STUB_ALWAYS_PASS `test rules` always exits 0, even when mutated
# STUB_WARN `test rules` emits promtool's unresolved-rule-file warning
# and still exits 0
mkdir -p "$TMP/bin"
cat > "$TMP/bin/promtool" <<'STUB'
#!/bin/sh
case "${1:-}" in
--version) echo "promtool, version ${STUB_VERSION:-3.15.0} (branch: stub)"; exit 0 ;;
esac
case "${1:-} ${2:-}" in
"check rules") echo " SUCCESS: 1 rules found"; exit 0 ;;
"test rules")
[ -n "${STUB_WARN:-}" ] && echo " WARNING: no file match pattern nowhere.yaml"
[ -n "${STUB_ALWAYS_PASS:-}" ] && { echo " SUCCESS"; exit 0; }
if grep -rq 'MUTATED-SENTINEL' . 2>/dev/null; then
echo " FAILED: sentinel present"; exit 1
fi
echo " SUCCESS"; exit 0 ;;
esac
echo "stub: unhandled [$*]" >&2; exit 2
STUB
chmod +x "$TMP/bin/promtool"
# --- fixtures -------------------------------------------------------------
# A whole fixture repo per case, so no case can leak into another. `rules` is
# the count of PrometheusRule manifests to emit; the suite references the first
# one via `../`, the same relative shape the real suites use.
mkfixture() { # <dir> [n-manifests]
d="$1"; n="${2:-1}"
mkdir -p "$d/kubernetes/apps/monitoring-rules/tests"
i=1
while [ "$i" -le "$n" ]; do
cat > "$d/kubernetes/apps/monitoring-rules/prometheusrule-fix$i.yaml" <<YAML
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: fix$i
spec:
groups:
- name: fix$i.rules
rules:
- alert: Fixture$i
expr: up == 0
for: 8m
YAML
i=$((i + 1))
done
cat > "$d/kubernetes/apps/monitoring-rules/tests/fixture_test.yaml" <<'YAML'
rule_files:
- ../prometheusrule-fix1.yaml
tests:
- interval: 1m
input_series:
- series: 'up'
values: '1x10'
alert_rule_test:
- eval_time: 10m
alertname: Fixture1
exp_alerts: []
YAML
# A control whose sed inserts the stub's sentinel, so the baseline passes and
# the mutant fails — the shape a real control has.
printf 'sentinel|kubernetes/apps/monitoring-rules/prometheusrule-fix1.yaml|s/for: 8m/for: 8m # MUTATED-SENTINEL/\n' \
> "$d/mutations.txt"
}
# run <fixture-dir> [env assignments...] -> exit status of the SUT, output in
# $TMP/out. Trailing assignments are appended AFTER the defaults, and `env`
# applies them in order, so a case can override any default.
run() {
d="$1"; shift
set +e
( cd "$d" && env PATH="$TMP/bin:$PATH" \
PROMTOOL_RULE_FLOOR=1 \
PROMTOOL_MUTATIONS=mutations.txt \
"$@" sh "$SUT" . ) > "$TMP/out" 2>&1
rc=$?
set -e
return $rc
}
# expect_fail <name> <fixture> <substring the output must contain> [env...]
#
# The substring is not decoration. A test that only asserts "exited non-zero"
# passes when the script fails for an unrelated reason — a typo'd fixture, a
# floor left set by an earlier case — and that is how a guard test rots into a
# test of nothing, which is the same disease this whole script exists to treat.
expect_fail() {
name="$1"; d="$2"; want="$3"; shift 3
if run "$d" "$@"; then
printf 'FAIL %s: expected non-zero exit, got 0\n' "$name"
sed 's/^/ /' "$TMP/out"
fails=$((fails + 1))
elif ! grep -qF "$want" "$TMP/out"; then
printf 'FAIL %s: rejected, but not for the expected reason\n' "$name"
printf ' wanted output containing: %s\n' "$want"
sed 's/^/ /' "$TMP/out"
fails=$((fails + 1))
else
printf 'ok %s\n' "$name"
fi
}
expect_pass() { # <name> <fixture> [env...]
name="$1"; d="$2"; shift 2
if run "$d" "$@"; then
printf 'ok %s\n' "$name"
else
printf 'FAIL %s: expected exit 0\n' "$name"
sed 's/^/ /' "$TMP/out"
fails=$((fails + 1))
fi
}
# --- happy path, first: if this does not pass, nothing below means anything --
mkfixture "$TMP/happy" 1
expect_pass "happy-path" "$TMP/happy"
# A green run must actually say it ran the mutation. Guards against the whole
# phase being skipped while the script still exits 0.
if ! grep -q 'mutation \[sentinel\] detected' "$TMP/out"; then
echo "FAIL happy-path did not report the mutation as detected"
sed 's/^/ /' "$TMP/out"
fails=$((fails + 1))
else
echo "ok happy-path reports the mutation as detected"
fi
# --- the floors -----------------------------------------------------------
mkdir -p "$TMP/empty/kubernetes"
: > "$TMP/empty/mutations.txt"
expect_fail "empty-discovery (0 manifests)" "$TMP/empty" \
"found only 0 PrometheusRule manifests"
mkfixture "$TMP/nosuites" 1
rm "$TMP/nosuites/kubernetes/apps/monitoring-rules/tests/fixture_test.yaml"
expect_fail "no-suites (0 test files)" "$TMP/nosuites" \
"found only 0 *_test.yaml suites"
mkfixture "$TMP/floor" 1
expect_fail "rule floor not met (1 < 5)" "$TMP/floor" \
"found only 1 PrometheusRule manifests; floor is 5" PROMTOOL_RULE_FLOOR=5
# --- the hole that shipped ------------------------------------------------
mkfixture "$TMP/unresolved" 1
sed 's|\.\./prometheusrule-fix1\.yaml|kubernetes/apps/monitoring-rules/prometheusrule-fix1.yaml|' \
"$TMP/unresolved/kubernetes/apps/monitoring-rules/tests/fixture_test.yaml" > "$TMP/u.tmp"
mv "$TMP/u.tmp" "$TMP/unresolved/kubernetes/apps/monitoring-rules/tests/fixture_test.yaml"
# The expected substring must sit on ONE line: die() messages are wrapped in the
# source, so a phrase spanning the wrap will not match.
expect_fail "unresolved-rule-file (repo-root path, not suite-relative)" "$TMP/unresolved" \
"references rule_files entry"
mkfixture "$TMP/warn" 1
expect_fail "warning-only (promtool WARNs but exits 0)" "$TMP/warn" \
"do not satisfy their own unit suites" STUB_WARN=1
mkfixture "$TMP/inline" 1
printf 'rule_files: [../prometheusrule-fix1.yaml]\ntests: []\n' \
> "$TMP/inline/kubernetes/apps/monitoring-rules/tests/fixture_test.yaml"
expect_fail "inline-rule-files" "$TMP/inline" "writes rule_files inline"
mkfixture "$TMP/norf" 1
printf 'tests: []\n' > "$TMP/norf/kubernetes/apps/monitoring-rules/tests/fixture_test.yaml"
expect_fail "empty-rule-files" "$TMP/norf" "declares no rule_files"
# --- a manifest the strip cannot turn into a rule file --------------------
mkfixture "$TMP/badspec" 2
cat > "$TMP/badspec/kubernetes/apps/monitoring-rules/prometheusrule-fix2.yaml" <<'YAML'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: fix2
# No top-level spec: at all. promtool check rules on the empty strip would
# report "SUCCESS: 0 rules found" and exit 0.
YAML
expect_fail "unstrippable-spec (no groups: after strip)" "$TMP/badspec" \
"expected [groups:]"
# --- the mutation phase's own guards -------------------------------------
mkfixture "$TMP/noop" 1
printf 'noop|kubernetes/apps/monitoring-rules/prometheusrule-fix1.yaml|s/for: 99m/for: 1m/\n' \
> "$TMP/noop/mutations.txt"
expect_fail "mutation-no-op (sed matches nothing)" "$TMP/noop" "changed nothing in"
mkfixture "$TMP/undetected" 1
expect_fail "mutation-undetected (suite passes while mutated)" "$TMP/undetected" \
"was NOT detected" STUB_ALWAYS_PASS=1
mkfixture "$TMP/nomut" 1
printf '# only a comment, no mutations\n' > "$TMP/nomut/mutations.txt"
expect_fail "mutation floor not met (0 mutations)" "$TMP/nomut" "ran only 0 mutations"
mkfixture "$TMP/mutmissing" 1
rm "$TMP/mutmissing/mutations.txt"
expect_fail "mutations file missing" "$TMP/mutmissing" "mutations.txt is missing"
mkfixture "$TMP/mutbadtarget" 1
printf 'bad|kubernetes/apps/monitoring-rules/does-not-exist.yaml|s/a/b/\n' \
> "$TMP/mutbadtarget/mutations.txt"
expect_fail "mutation targets an undiscovered manifest" "$TMP/mutbadtarget" \
"which is not one of"
# --- the version pin ------------------------------------------------------
mkfixture "$TMP/v2" 1
expect_fail "promtool 2.x" "$TMP/v2" "is not 3.x" STUB_VERSION=2.55.1
mkfixture "$TMP/vjunk" 1
expect_fail "unparseable promtool version" "$TMP/vjunk" \
"cannot parse" STUB_VERSION=not-a-version
if [ "$fails" -eq 0 ]; then
echo "PASS all promtool-rules.sh guard tests"
exit 0
fi
echo "FAIL $fails promtool-rules.sh guard test(s)"
exit 1