Refs #326. That issue is the alert-monitor suppression state and stays open by design — this fixes the MECHANISM that maintains it, not the issue.
Two defects, both of which silently mute real alerts.
1. Fingerprints mentioned in prose became suppressions
The dedup step grepped the ENTIRE issue body for 16-hex tokens, so triage reasons that cite another alert fingerprint in prose ("see the issue filed for fingerprint X") were read as suppression entries.
Verified against the live noise log: the old extractor yields 22 fingerprints for 21 real bullets. The extra one is 2ddc41edb689fc33 — the crowdsec alert that HAD been filed as a genuine issue, permanently muted purely because two other entries mention it in their reason text.
Now only lines in the exact shape the write-back step emits count:
- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>
Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among them.
2. Suppression never expired
The fingerprint is sha256(alertname|scope) and is deliberately stable across recurrences. Combined with a permanent noise log, one blip verdict mutes that alertname+scope pair FOREVER.
21 pairs are currently muted that way, including KubePodNotReady, KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods for openbao — all recorded as "node reboot evicted openbao-0, self-resolved". As it stands, a real openbao outage would file no issue at all.
A noise verdict means "this occurrence was benign", not "this alert can never be interesting again". Entries now expire after NOISE_TTL_DAYS (default 30), counted from the date already in each line. If it is still noise, triage re-adds it automatically.
Rollout is gentle by construction
Every current entry is dated 2026-07-21 or later, so at a 30-day TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge. The oldest entries lapse a day later, and only get re-triaged if they actually fire again.
Unparseable lines are dropped — they stop suppressing. This fails toward MORE alerting, never less.
Verification
Ran the exact shipped snippet inside the real ci-claude:2.1.218 image against the live noise log:
Expiry exercised at several cutoffs: 2026-07-20 -> 21 kept, 2026-07-25 -> 5 kept.
Worth knowing: that image ships BusyBox date, not GNU — date -d "30 days ago" fails there outright. The cutoff therefore uses epoch arithmetic plus date -d @<epoch>, confirmed working in that image. BusyBox sed -E and awk gsub were confirmed the same way, by running the snippet in-image rather than trusting local GNU tools.
Refs #326. That issue is the alert-monitor suppression state and stays open by design — this fixes the MECHANISM that maintains it, not the issue.
Two defects, both of which silently mute real alerts.
## 1. Fingerprints mentioned in prose became suppressions
The dedup step grepped the ENTIRE issue body for 16-hex tokens, so triage reasons that cite another alert fingerprint in prose ("see the issue filed for fingerprint X") were read as suppression entries.
Verified against the live noise log: the old extractor yields **22** fingerprints for **21** real bullets. The extra one is `2ddc41edb689fc33` — the crowdsec alert that HAD been filed as a genuine issue, permanently muted purely because two other entries mention it in their reason text.
Now only lines in the exact shape the write-back step emits count:
```
- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>
```
Re-verified with the new parser: **21**, and `2ddc41edb689fc33` is no longer among them.
## 2. Suppression never expired
The fingerprint is `sha256(alertname|scope)` and is deliberately stable across recurrences. Combined with a permanent noise log, one blip verdict mutes that alertname+scope pair FOREVER.
21 pairs are currently muted that way, including `KubePodNotReady`, `KubeStatefulSetReplicasMismatch` and `KubePdbNotEnoughHealthyPods` for **openbao** — all recorded as "node reboot evicted openbao-0, self-resolved". As it stands, a real openbao outage would file no issue at all.
A noise verdict means "this occurrence was benign", not "this alert can never be interesting again". Entries now expire after `NOISE_TTL_DAYS` (default 30), counted from the date already in each line. If it is still noise, triage re-adds it automatically.
### Rollout is gentle by construction
Every current entry is dated `2026-07-21` or later, so at a 30-day TTL the cutoff is `2026-07-20` and NOTHING un-mutes on merge. The oldest entries lapse a day later, and only get re-triaged if they actually fire again.
Unparseable lines are dropped — they stop suppressing. This fails toward MORE alerting, never less.
## Verification
Ran the exact shipped snippet inside the real `ci-claude:2.1.218` image against the live noise log:
```
cutoff=2026-07-20
suppressions=21
prose-only fp 2ddc41edb689fc33 present? false
```
Expiry exercised at several cutoffs: `2026-07-20` -> 21 kept, `2026-07-25` -> 5 kept.
**Worth knowing:** that image ships BusyBox `date`, not GNU — `date -d "30 days ago"` fails there outright. The cutoff therefore uses epoch arithmetic plus `date -d @<epoch>`, confirmed working in that image. BusyBox `sed -E` and `awk gsub` were confirmed the same way, by running the snippet in-image rather than trusting local GNU tools.
`woodpecker lint` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Two defects in the noise-log dedup, both of which silently mute real alerts.
1. Prose-mentioned fingerprints became suppressions.
The extractor grepped the WHOLE issue body for 16-hex tokens. Triage reasons
routinely cite another alert's fingerprint in prose ("see the issue filed for
fingerprint <fp>"), so those citations were read as suppression entries.
Verified against the live noise log: the old extractor yields 22 fingerprints
for 21 real entries. The extra one is 2ddc41edb689fc33 -- the crowdsec alert
that HAD been filed as a genuine issue, permanently muted purely because two
other entries mention it. Now only lines in the exact shape the write-back step
emits count:
- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>
Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among
them.
2. Suppression never expired.
The fingerprint is sha256(alertname|scope) and is deliberately stable across
recurrences, so a single blip verdict muted that alertname+scope pair forever.
21 pairs were permanently muted, including openbao's KubePodNotReady,
KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods -- so a real
openbao outage would have filed no issue at all.
A noise verdict means "this occurrence was benign", not "this alert can never be
interesting again". Entries now expire after NOISE_TTL_DAYS (default 30) counted
from the date already present in each line. Re-triaging a recurrence weeks later
is cheap; missing an outage because a July blip muted it is not. If it is still
noise, triage re-adds it automatically.
Rollout is gentle by construction: every current entry is dated 2026-07-21 or
later, so at a 30d TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge --
the oldest entries lapse a day later, and only re-triage if they actually fire
again.
Unparseable lines are dropped, i.e. they stop suppressing: this fails toward
more alerting, never less.
Verified end-to-end by running the shipped snippet inside the real
ci-claude:2.1.218 image against the live noise log: cutoff 2026-07-20,
21 suppressions, prose-only fingerprint absent. Note that image ships BusyBox
date, not GNU -- `date -d '30 days ago'` errors there, so the cutoff uses epoch
arithmetic with `date -d @<epoch>`, which was confirmed working in that image.
Refs #326
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🤖Automated review — model claude-sonnet-5 (complexity: normal), commit 8a8145011a54d4f2de6db097aba48cc9b8081c12.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR fixes two real defects in .woodpecker/alert-monitor.yml's noise-log suppression logic: (1) the fingerprint extractor used to grep the entire noise-log issue body for any 16-hex token, so a fingerprint merely mentioned in prose (e.g. "see the issue filed for fingerprint X") was silently treated as a suppression; (2) suppressions never expired, so a single "blip, self-resolved" verdict permanently muted that alertname+scope pair. I traced the new sed/awk pipeline against the exact write-back format (.woodpecker/alert-monitor.yml:279) by hand and it parses correctly: it only matches lines shaped exactly like - <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>, a bare fingerprint in reason prose is correctly ignored, and the TTL cutoff comparison (date→YYYYMMDD integer compare) is correct. I also confirmed byte-for-byte that the em dash (U+2014, e2 80 94) used in the new regex (line 165) and the one emitted by the existing printf at line 279 are identical, so the anchor character matches. No blocking issues found.
🟡 Suggestions
No automated test for the new parsing logic (.woodpecker/alert-monitor.yml:162-169). This repo has an established pattern for exactly this situation — scripts/ci/test-model-selection.sh + .woodpecker/ci-scripts-test.yml — extracting nontrivial embedded CI shell/regex logic into a standalone script with an offline unit test that runs on every PR. This change adds a sed -nE/awk regex-and-date-arithmetic pipeline of similar complexity (and similar blast radius — a subtly wrong regex here silently drops or keeps suppressions) but leaves it inline in the YAML with only manual, non-repeatable verification described in the PR body. Extracting it to scripts/ci/ with a test using canned noise-log bodies (a prose-only fingerprint mention, an expired entry, a boundary-date entry) would catch a future regression the way test-model-selection.sh does for pick-model.sh.
Noise log grows unboundedly and can accumulate duplicate fingerprint lines (.woodpecker/alert-monitor.yml:279-291). Expiry only affects which fingerprints are treated as currently suppressing — expired lines are never removed from the issue body itself. If a fingerprint's suppression expires and it's later re-triaged as noise again, a second line for the same fp/alertname gets appended rather than the old one being replaced, so the body accumulates stale, superseded entries indefinitely. This doesn't cause incorrect behavior (a duplicate fp in the FPS array is harmless for the IN(...) membership check), but it means the "delete a line to un-suppress" maintenance instructions in the header (line 293) become less reliable over time since there could be multiple lines for the same alert. Worth a follow-up to prune/replace on write-back, or at least noting it's out of scope.
🟢 Nits
The BusyBox date -d @<epoch> behavior (.woodpecker/alert-monitor.yml:163) is confirmed working per the PR description's in-image verification, but that verification isn't captured as a repo-checked-in test (ties into the suggestion above) — if the ci-claude base image ever changes its date/busybox build, this would fail silently only at pipeline-run time.
Tests
No automated tests are added. The PR body documents thorough manual verification (fingerprint counts before/after, TTL cutoff behavior at two dates, BusyBox-specific tool behavior) run against the live noise log inside the actual ci-claude:2.1.218 image, which is credible and specific. I independently re-derived the regex/awk behavior against synthetic noise-log lines matching the real write-back format and confirmed it distinguishes a fingerprint mentioned only in prose from one in a genuine suppression line, and correctly ages out an entry past the TTL cutoff. Given this script isn't easily exercised by this review's checkout (no live Gitea issue, no cluster), the manual verification is a reasonable substitute for merge, but see the suggestion above about codifying it as a regression test.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `8a8145011a54d4f2de6db097aba48cc9b8081c12`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR fixes two real defects in `.woodpecker/alert-monitor.yml`'s noise-log suppression logic: (1) the fingerprint extractor used to grep the entire noise-log issue body for any 16-hex token, so a fingerprint merely *mentioned in prose* (e.g. "see the issue filed for fingerprint X") was silently treated as a suppression; (2) suppressions never expired, so a single "blip, self-resolved" verdict permanently muted that `alertname+scope` pair. I traced the new `sed`/`awk` pipeline against the exact write-back format (`.woodpecker/alert-monitor.yml:279`) by hand and it parses correctly: it only matches lines shaped exactly like `- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>`, a bare fingerprint in reason prose is correctly ignored, and the TTL cutoff comparison (date→`YYYYMMDD` integer compare) is correct. I also confirmed byte-for-byte that the em dash (`U+2014`, `e2 80 94`) used in the new regex (line 165) and the one emitted by the existing `printf` at line 279 are identical, so the anchor character matches. No blocking issues found.
## 🟡 Suggestions
- **No automated test for the new parsing logic** (`.woodpecker/alert-monitor.yml:162-169`). This repo has an established pattern for exactly this situation — `scripts/ci/test-model-selection.sh` + `.woodpecker/ci-scripts-test.yml` — extracting nontrivial embedded CI shell/regex logic into a standalone script with an offline unit test that runs on every PR. This change adds a `sed -nE`/`awk` regex-and-date-arithmetic pipeline of similar complexity (and similar blast radius — a subtly wrong regex here silently drops or keeps suppressions) but leaves it inline in the YAML with only manual, non-repeatable verification described in the PR body. Extracting it to `scripts/ci/` with a test using canned noise-log bodies (a prose-only fingerprint mention, an expired entry, a boundary-date entry) would catch a future regression the way `test-model-selection.sh` does for `pick-model.sh`.
- **Noise log grows unboundedly and can accumulate duplicate fingerprint lines** (`.woodpecker/alert-monitor.yml:279-291`). Expiry only affects which fingerprints are treated as *currently suppressing* — expired lines are never removed from the issue body itself. If a fingerprint's suppression expires and it's later re-triaged as noise again, a second line for the same fp/alertname gets appended rather than the old one being replaced, so the body accumulates stale, superseded entries indefinitely. This doesn't cause incorrect behavior (a duplicate fp in the `FPS` array is harmless for the `IN(...)` membership check), but it means the "delete a line to un-suppress" maintenance instructions in the header (line 293) become less reliable over time since there could be multiple lines for the same alert. Worth a follow-up to prune/replace on write-back, or at least noting it's out of scope.
## 🟢 Nits
- The BusyBox `date -d @<epoch>` behavior (`.woodpecker/alert-monitor.yml:163`) is confirmed working per the PR description's in-image verification, but that verification isn't captured as a repo-checked-in test (ties into the suggestion above) — if the `ci-claude` base image ever changes its `date`/`busybox` build, this would fail silently only at pipeline-run time.
## Tests
No automated tests are added. The PR body documents thorough manual verification (fingerprint counts before/after, TTL cutoff behavior at two dates, BusyBox-specific tool behavior) run against the live noise log inside the actual `ci-claude:2.1.218` image, which is credible and specific. I independently re-derived the regex/awk behavior against synthetic noise-log lines matching the real write-back format and confirmed it distinguishes a fingerprint mentioned only in prose from one in a genuine suppression line, and correctly ages out an entry past the TTL cutoff. Given this script isn't easily exercised by this review's checkout (no live Gitea issue, no cluster), the manual verification is a reasonable substitute for merge, but see the suggestion above about codifying it as a regression test.
## Recommendation
Looks reasonable — no blocking issues; comments above are optional.
<!-- pr-review:sha=8a8145011a54d4f2de6db097aba48cc9b8081c12 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
Refs #326. That issue is the alert-monitor suppression state and stays open by design — this fixes the MECHANISM that maintains it, not the issue.
Two defects, both of which silently mute real alerts.
1. Fingerprints mentioned in prose became suppressions
The dedup step grepped the ENTIRE issue body for 16-hex tokens, so triage reasons that cite another alert fingerprint in prose ("see the issue filed for fingerprint X") were read as suppression entries.
Verified against the live noise log: the old extractor yields 22 fingerprints for 21 real bullets. The extra one is
2ddc41edb689fc33— the crowdsec alert that HAD been filed as a genuine issue, permanently muted purely because two other entries mention it in their reason text.Now only lines in the exact shape the write-back step emits count:
Re-verified with the new parser: 21, and
2ddc41edb689fc33is no longer among them.2. Suppression never expired
The fingerprint is
sha256(alertname|scope)and is deliberately stable across recurrences. Combined with a permanent noise log, one blip verdict mutes that alertname+scope pair FOREVER.21 pairs are currently muted that way, including
KubePodNotReady,KubeStatefulSetReplicasMismatchandKubePdbNotEnoughHealthyPodsfor openbao — all recorded as "node reboot evicted openbao-0, self-resolved". As it stands, a real openbao outage would file no issue at all.A noise verdict means "this occurrence was benign", not "this alert can never be interesting again". Entries now expire after
NOISE_TTL_DAYS(default 30), counted from the date already in each line. If it is still noise, triage re-adds it automatically.Rollout is gentle by construction
Every current entry is dated
2026-07-21or later, so at a 30-day TTL the cutoff is2026-07-20and NOTHING un-mutes on merge. The oldest entries lapse a day later, and only get re-triaged if they actually fire again.Unparseable lines are dropped — they stop suppressing. This fails toward MORE alerting, never less.
Verification
Ran the exact shipped snippet inside the real
ci-claude:2.1.218image against the live noise log:Expiry exercised at several cutoffs:
2026-07-20-> 21 kept,2026-07-25-> 5 kept.Worth knowing: that image ships BusyBox
date, not GNU —date -d "30 days ago"fails there outright. The cutoff therefore uses epoch arithmetic plusdate -d @<epoch>, confirmed working in that image. BusyBoxsed -Eandawk gsubwere confirmed the same way, by running the snippet in-image rather than trusting local GNU tools.woodpecker lintpasses.🤖 Generated with Claude Code
Two defects in the noise-log dedup, both of which silently mute real alerts. 1. Prose-mentioned fingerprints became suppressions. The extractor grepped the WHOLE issue body for 16-hex tokens. Triage reasons routinely cite another alert's fingerprint in prose ("see the issue filed for fingerprint <fp>"), so those citations were read as suppression entries. Verified against the live noise log: the old extractor yields 22 fingerprints for 21 real entries. The extra one is 2ddc41edb689fc33 -- the crowdsec alert that HAD been filed as a genuine issue, permanently muted purely because two other entries mention it. Now only lines in the exact shape the write-back step emits count: - <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason> Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among them. 2. Suppression never expired. The fingerprint is sha256(alertname|scope) and is deliberately stable across recurrences, so a single blip verdict muted that alertname+scope pair forever. 21 pairs were permanently muted, including openbao's KubePodNotReady, KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods -- so a real openbao outage would have filed no issue at all. A noise verdict means "this occurrence was benign", not "this alert can never be interesting again". Entries now expire after NOISE_TTL_DAYS (default 30) counted from the date already present in each line. Re-triaging a recurrence weeks later is cheap; missing an outage because a July blip muted it is not. If it is still noise, triage re-adds it automatically. Rollout is gentle by construction: every current entry is dated 2026-07-21 or later, so at a 30d TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge -- the oldest entries lapse a day later, and only re-triage if they actually fire again. Unparseable lines are dropped, i.e. they stop suppressing: this fails toward more alerting, never less. Verified end-to-end by running the shipped snippet inside the real ci-claude:2.1.218 image against the live noise log: cutoff 2026-07-20, 21 suppressions, prose-only fingerprint absent. Note that image ships BusyBox date, not GNU -- `date -d '30 days ago'` errors there, so the cutoff uses epoch arithmetic with `date -d @<epoch>`, which was confirmed working in that image. Refs #326 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit8a8145011a54d4f2de6db097aba48cc9b8081c12.Summary
This PR fixes two real defects in
.woodpecker/alert-monitor.yml's noise-log suppression logic: (1) the fingerprint extractor used to grep the entire noise-log issue body for any 16-hex token, so a fingerprint merely mentioned in prose (e.g. "see the issue filed for fingerprint X") was silently treated as a suppression; (2) suppressions never expired, so a single "blip, self-resolved" verdict permanently muted thatalertname+scopepair. I traced the newsed/awkpipeline against the exact write-back format (.woodpecker/alert-monitor.yml:279) by hand and it parses correctly: it only matches lines shaped exactly like- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>, a bare fingerprint in reason prose is correctly ignored, and the TTL cutoff comparison (date→YYYYMMDDinteger compare) is correct. I also confirmed byte-for-byte that the em dash (U+2014,e2 80 94) used in the new regex (line 165) and the one emitted by the existingprintfat line 279 are identical, so the anchor character matches. No blocking issues found.🟡 Suggestions
.woodpecker/alert-monitor.yml:162-169). This repo has an established pattern for exactly this situation —scripts/ci/test-model-selection.sh+.woodpecker/ci-scripts-test.yml— extracting nontrivial embedded CI shell/regex logic into a standalone script with an offline unit test that runs on every PR. This change adds ased -nE/awkregex-and-date-arithmetic pipeline of similar complexity (and similar blast radius — a subtly wrong regex here silently drops or keeps suppressions) but leaves it inline in the YAML with only manual, non-repeatable verification described in the PR body. Extracting it toscripts/ci/with a test using canned noise-log bodies (a prose-only fingerprint mention, an expired entry, a boundary-date entry) would catch a future regression the waytest-model-selection.shdoes forpick-model.sh..woodpecker/alert-monitor.yml:279-291). Expiry only affects which fingerprints are treated as currently suppressing — expired lines are never removed from the issue body itself. If a fingerprint's suppression expires and it's later re-triaged as noise again, a second line for the same fp/alertname gets appended rather than the old one being replaced, so the body accumulates stale, superseded entries indefinitely. This doesn't cause incorrect behavior (a duplicate fp in theFPSarray is harmless for theIN(...)membership check), but it means the "delete a line to un-suppress" maintenance instructions in the header (line 293) become less reliable over time since there could be multiple lines for the same alert. Worth a follow-up to prune/replace on write-back, or at least noting it's out of scope.🟢 Nits
date -d @<epoch>behavior (.woodpecker/alert-monitor.yml:163) is confirmed working per the PR description's in-image verification, but that verification isn't captured as a repo-checked-in test (ties into the suggestion above) — if theci-claudebase image ever changes itsdate/busyboxbuild, this would fail silently only at pipeline-run time.Tests
No automated tests are added. The PR body documents thorough manual verification (fingerprint counts before/after, TTL cutoff behavior at two dates, BusyBox-specific tool behavior) run against the live noise log inside the actual
ci-claude:2.1.218image, which is credible and specific. I independently re-derived the regex/awk behavior against synthetic noise-log lines matching the real write-back format and confirmed it distinguishes a fingerprint mentioned only in prose from one in a genuine suppression line, and correctly ages out an entry past the TTL cutoff. Given this script isn't easily exercised by this review's checkout (no live Gitea issue, no cluster), the manual verification is a reasonable substitute for merge, but see the suggestion above about codifying it as a regression test.Recommendation
Looks reasonable — no blocking issues; comments above are optional.