feat(alert-monitor): blip lookback — catch alerts that resolve between ticks #269

2026-07-16T18:50:11Z

claude-bot commented

2026-07-16 18:50:11 +00:00

The owner noticed blips the monitor never saw. Verified from Prometheus ALERTS history: yesterday evening's 4-alert home-assistant incident (18:46–23:11Z) and this morning's 17-minute KubeHpaMaxedOut (authentik, 06:29–06:46Z) — both invisible to a 15-min live-snapshot poll (the HA case predates the monitor; the HPA blip is the class that would ALWAYS escape).

Fetch step: additionally queries max_over_time(ALERTS{alertstate=\"firing\"}[35m]) via the Prometheus service proxy (fallback to empty on failure; no jq in ci-k8s so counting stays in the agent step).
Dedup step: synthesizes pseudo-alerts (blip: true) for lookback series not currently firing at all — checked against the PRE-filter snapshot, so the live null-routed KubeCPUOvercommit does not reappear as a blip. Pseudo-fingerprint = sha256(alertname|scope)[:16], stable across recurrences → one noise verdict permanently suppresses a flapper via the existing noise log.
Blips then ride the normal title/noise dedup + triage. Prompt documents their semantics: self-resolved, judge the underlying cause; null-routed per the repo route config = noise; exact firing window available via query_range.

Tested against live data (see commit message). Lint green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

The owner noticed blips the monitor never saw. Verified from Prometheus ALERTS history: yesterday evening's 4-alert home-assistant incident (18:46–23:11Z) and this morning's 17-minute KubeHpaMaxedOut (authentik, 06:29–06:46Z) — both invisible to a 15-min live-snapshot poll (the HA case predates the monitor; the HPA blip is the class that would ALWAYS escape). - Fetch step: additionally queries `max_over_time(ALERTS{alertstate=\"firing\"}[35m])` via the Prometheus service proxy (fallback to empty on failure; no jq in ci-k8s so counting stays in the agent step). - Dedup step: synthesizes pseudo-alerts (`blip: true`) for lookback series not currently firing at all — checked against the PRE-filter snapshot, so the live null-routed KubeCPUOvercommit does not reappear as a blip. Pseudo-fingerprint = sha256(alertname|scope)[:16], stable across recurrences → one noise verdict permanently suppresses a flapper via the existing noise log. - Blips then ride the normal title/noise dedup + triage. Prompt documents their semantics: self-resolved, judge the underlying cause; null-routed per the repo route config = noise; exact firing window available via query_range. Tested against live data (see commit message). Lint green. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

claude-bot added 1 commit 2026-07-16 18:50:11 +00:00

feat(alert-monitor): blip lookback — catch alerts that resolve between ticks c10c7af3c8

The live 15-min snapshot structurally misses anything that fires and
resolves between ticks: on 2026-07-16 a 17-minute KubeHpaMaxedOut
(authentik) and last evening's four-alert home-assistant incident were
invisible to the monitor (owner noticed the blips; monitor did not).

The fetch step now also queries Prometheus's ALERTS history over a
35-minute window (~2 ticks + jitter). The dedup step synthesizes
pseudo-alerts for series not currently firing at all — checked against
the PRE-filter snapshot so live-but-null-routed alerts don't reappear
as blips — with a stable sha256-derived fingerprint so one noise
verdict suppresses a flappy alert permanently. Blips then flow through
the same title/noise dedup and triage as live alerts; the prompt
explains their semantics (self-resolved; judge the underlying cause;
null-routed per the repo route config = noise).

Verified against live data: currently-firing KubeCPUOvercommit
excluded, injected KubeHpaMaxedOut blip synthesized correctly;
sha256sum confirmed present in ci-claude; lookback query tested via
the service proxy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei

k.renner merged commit e5a5c48431 into master

2026-07-16 19:00:56 +00:00

k.renner referenced this issue from a commit

2026-07-16 19:00:58 +00:00

Merge pull request 'feat(alert-monitor): blip lookback — catch alerts that resolve between ticks' (#269) from feat/alert-monitor-blip-lookback into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#269