This PR does not fix the firing alert — the alert is not repo-fixable.Alertmanager*FailedToSendAlerts is firing because Proton's SMTP submission endpoint is rejecting Alertmanager's AUTH dialogue. The credential itself lives in OpenBao (apps/monitoring/alertmanager-smtp), not in this repository, so no manifest change can resolve it; the remediation is a human action described under Manual action required below. The only change here is a documentation correction: the Alertmanager comment block in helmrelease-prometheus.yaml still claimed the SMTP token lives in a "SOPS-encrypted Secret", which sends whoever responds to this exact incident to a file that does not exist. That comment is now corrected to point at OpenBao and the ExternalSecret.
I am flagging two places where the issue's analysis did not match what I observed live this run — see Notes.
Changes
kubernetes/apps/monitoring/helmrelease-prometheus.yaml — corrected the stale comment above config:. The SMTP token is no longer in a SOPS-encrypted Secret; it is stored in OpenBao at apps/monitoring/alertmanager-smtp and projected into the alertmanager-smtp Secret by alertmanager-smtp-externalsecret.yaml. Comment-only; no rendered manifest change.
Verification
Observed live this run (all read-only, via ./.ci-diag/kubectl):
kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager → 2/2 Running, 0 restarts, age 5h26m. Alertmanager itself is healthy; this is not a pod/rollout problem.
kubectl logs alertmanager-...-0 -n monitoring -c alertmanager --tail=30 → the error has changed since the issue was written. Up to 23:28:04Z it is the reported 454; from 23:28:14Z onward it is a different code:
Prometheus API proxy, increase(alertmanager_notifications_total{integration="email"}[1h]) → 20.2; increase(alertmanager_notifications_failed_total{integration="email"}[1h]) → other = 16.0, all other reasons (authError, rateLimited, serverError, clientError) = 0. So roughly 4 of 20 sends in the last hour got through.
kubectl get externalsecret -n monitoring alertmanager-smtp → Ready True SecretSynced, refreshTime=2026-08-26T23:34:25Z. Secret delivery into the cluster is working; this part of the issue's analysis holds.
Repo-side verification of the comment fix:
kubernetes/apps/monitoring/alertmanager-smtp-externalsecret.yaml sources apps/monitoring/alertmanager-smtp from the openbao ClusterSecretStore into a Secret named alertmanager-smtp.
find kubernetes -name '*sops*' returns only kubernetes/.sops.yaml and kubernetes/MIGRATION-sops-openbao.md — there is no SOPS-encrypted alertmanager secret, confirming the old comment was stale.
Reviewer check: this change is comment-only, so after Flux reconciles the rendered Alertmanager config must be byte-identical. Confirm the HelmRelease reconciles cleanly and that no Alertmanager restart or config reload is triggered. No alert will stop firing as a result of this PR.
Manual action required (not doable from the repo)
Check https://status.proton.me. The onset error (454 4.7.0 ... Connection lost to authentication server) is emitted when Proton's SMTP frontend cannot reach its own auth backend.
The issue set its own escalation trigger — "if the 454s ... turn into 535, regenerate the Proton SMTP token". That has now happened. However, the intermittent successes above are evidence against a fully revoked token: a rejected credential returns 535 on every attempt, and this one is being accepted several times an hour. Before rotating, re-check whether the 535s have stopped, since rotation is disruptive and may be treating a provider-side symptom.
If failures persist without any successes, regenerate the token in the Proton webapp (Settings → IMAP/SMTP) and write it to OpenBao at apps/monitoring/alertmanager-smtp. ESO refreshes hourly (refreshInterval: 1h), so force a resync or wait for the next refresh.
Notes
Two corrections to the issue, both grounded in output observed this run:
"Outbound alerting is currently blind" / "total alerting blackout" is not accurate as of this run. Delivery is degraded, not dead — about 4 of 20 sends succeeded in the last hour, and the AlertmanagerFailedToSendAlerts notification itself was delivered at 23:30:38Z. Failures come in bursts: a 10-minute window ending ~23:44Z had zero successes, while 23:29–23:32Z had four. This lowers the severity but does mean some notifications are still being dropped after retry exhaustion.
The issue's first "ruled out" argument is unsound. It cites authError = 0 as proof the credential is fine. Alertmanager buckets SMTP authentication failures under reason="other", not reason="authError" — which matches exactly what I measured (other = 16, authError = 0) while the logs show nothing but auth errors. So that metric cannot rule a credential problem in or out. The real evidence that the credential is still valid is the intermittent Notify success, not the metric. The conclusion (provider-side fault) is still the best-supported reading, but not for the reason given.
Deliberately not done:
No fallback receiver added, despite the issue suggesting one. Adding an ntfy/Gotify/webhook receiver means introducing a new external dependency and an endpoint that does not exist in this cluster today; a receiver pointing at a non-existent target would fail silently and is worse than the current single channel. This is a real gap worth addressing, but it is a design change for a human to scope, not a surgical alert fix.
No retry/threshold tuning and no alert silencing. Alertmanager exposes no max_retries knob for email, and suppressing Alertmanager*FailedToSendAlerts would hide precisely the signal that outbound alerting is broken.
The issue contained no "Verified owner guidance" section, so everything in it was treated as an unverified hypothesis and checked against live state.
## What this PR does
**This PR does not fix the firing alert — the alert is not repo-fixable.** `Alertmanager*FailedToSendAlerts` is firing because Proton's SMTP submission endpoint is rejecting Alertmanager's AUTH dialogue. The credential itself lives in OpenBao (`apps/monitoring/alertmanager-smtp`), not in this repository, so no manifest change can resolve it; the remediation is a human action described under **Manual action required** below. The only change here is a documentation correction: the Alertmanager comment block in `helmrelease-prometheus.yaml` still claimed the SMTP token lives in a "SOPS-encrypted Secret", which sends whoever responds to this exact incident to a file that does not exist. That comment is now corrected to point at OpenBao and the ExternalSecret.
I am flagging two places where the issue's analysis did not match what I observed live this run — see **Notes**.
## Changes
- `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` — corrected the stale comment above `config:`. The SMTP token is no longer in a SOPS-encrypted Secret; it is stored in OpenBao at `apps/monitoring/alertmanager-smtp` and projected into the `alertmanager-smtp` Secret by `alertmanager-smtp-externalsecret.yaml`. Comment-only; no rendered manifest change.
## Verification
Observed live this run (all read-only, via `./.ci-diag/kubectl`):
- `kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager` → `2/2 Running`, `0` restarts, age `5h26m`. Alertmanager itself is healthy; this is not a pod/rollout problem.
- `kubectl logs alertmanager-...-0 -n monitoring -c alertmanager --tail=30` → the error **has changed since the issue was written**. Up to `23:28:04Z` it is the reported 454; from `23:28:14Z` onward it is a different code:
```
err="*smtp.plainAuth auth: 535 \"5.7.8 Error: authentication failed: authentication failed\""
```
- Same log tail → notifications are **partially succeeding**, e.g.:
```
23:29:49 level=INFO msg="Notify success" receiver=email attempts=14 numAlerts=1
23:30:38 level=INFO msg="Notify success" receiver=email attempts=8 numAlerts=1
23:31:55 level=INFO msg="Notify success" receiver=email attempts=14 numAlerts=2
```
- Prometheus API proxy, `increase(alertmanager_notifications_total{integration="email"}[1h])` → `20.2`; `increase(alertmanager_notifications_failed_total{integration="email"}[1h])` → `other = 16.0`, all other reasons (`authError`, `rateLimited`, `serverError`, `clientError`) = `0`. So roughly 4 of 20 sends in the last hour got through.
- `kubectl get externalsecret -n monitoring alertmanager-smtp` → `Ready True SecretSynced`, `refreshTime=2026-08-26T23:34:25Z`. Secret delivery into the cluster is working; this part of the issue's analysis holds.
Repo-side verification of the comment fix:
- `kubernetes/apps/monitoring/alertmanager-smtp-externalsecret.yaml` sources `apps/monitoring/alertmanager-smtp` from the `openbao` ClusterSecretStore into a Secret named `alertmanager-smtp`.
- `find kubernetes -name '*sops*'` returns only `kubernetes/.sops.yaml` and `kubernetes/MIGRATION-sops-openbao.md` — there is no SOPS-encrypted alertmanager secret, confirming the old comment was stale.
Reviewer check: this change is comment-only, so after Flux reconciles the rendered Alertmanager config must be byte-identical. Confirm the HelmRelease reconciles cleanly and that no Alertmanager restart or config reload is triggered. **No alert will stop firing as a result of this PR.**
## Manual action required (not doable from the repo)
1. Check <https://status.proton.me>. The onset error (`454 4.7.0 ... Connection lost to authentication server`) is emitted when Proton's SMTP frontend cannot reach its own auth backend.
2. The issue set its own escalation trigger — "if the 454s ... turn into `535`, regenerate the Proton SMTP token". **That has now happened.** However, the intermittent successes above are evidence *against* a fully revoked token: a rejected credential returns `535` on every attempt, and this one is being accepted several times an hour. Before rotating, re-check whether the `535`s have stopped, since rotation is disruptive and may be treating a provider-side symptom.
3. If failures persist without any successes, regenerate the token in the Proton webapp (Settings → IMAP/SMTP) and write it to OpenBao at `apps/monitoring/alertmanager-smtp`. ESO refreshes hourly (`refreshInterval: 1h`), so force a resync or wait for the next refresh.
## Notes
Two corrections to the issue, both grounded in output observed this run:
- **"Outbound alerting is currently blind" / "total alerting blackout" is not accurate as of this run.** Delivery is degraded, not dead — about 4 of 20 sends succeeded in the last hour, and the `AlertmanagerFailedToSendAlerts` notification itself was delivered at `23:30:38Z`. Failures come in bursts: a 10-minute window ending ~`23:44Z` had zero successes, while `23:29–23:32Z` had four. This lowers the severity but does mean some notifications are still being dropped after retry exhaustion.
- **The issue's first "ruled out" argument is unsound.** It cites `authError = 0` as proof the credential is fine. Alertmanager buckets SMTP authentication failures under `reason="other"`, not `reason="authError"` — which matches exactly what I measured (`other = 16`, `authError = 0`) while the logs show nothing but auth errors. So that metric cannot rule a credential problem in or out. The real evidence that the credential is still valid is the intermittent `Notify success`, not the metric. The conclusion (provider-side fault) is still the best-supported reading, but not for the reason given.
Deliberately **not** done:
- **No fallback receiver added,** despite the issue suggesting one. Adding an ntfy/Gotify/webhook receiver means introducing a new external dependency and an endpoint that does not exist in this cluster today; a receiver pointing at a non-existent target would fail silently and is worse than the current single channel. This is a real gap worth addressing, but it is a design change for a human to scope, not a surgical alert fix.
- **No retry/threshold tuning and no alert silencing.** Alertmanager exposes no `max_retries` knob for email, and suppressing `Alertmanager*FailedToSendAlerts` would hide precisely the signal that outbound alerting is broken.
- The issue contained no "Verified owner guidance" section, so everything in it was treated as an unverified hypothesis and checked against live state.
Closes #649
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 7e825aff0fe1acb3c6ab8b3dadd4ee6da3017d12.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR corrects a stale comment in the Alertmanager HelmRelease that incorrectly claimed the SMTP credential lives in a SOPS-encrypted Secret. The change is purely documentation—the comment now accurately points to OpenBao and the ExternalSecret that actually manages the credential. This fixes a real source of confusion for anyone debugging the AlertmanagerClusterFailedToSendAlerts alert, as the old comment would send them to a file that no longer exists.
🟢 Nits
None. The comment correction is accurate, minimal, and helpful. No YAML structure changes, no behavioral impact.
Tests
Not applicable—this is a comment-only change. The rendered Alertmanager config will be byte-identical after Flux reconciles.
Verification
Verified that:
alertmanager-smtp-externalsecret.yaml correctly sources the credential from OpenBao (apps/monitoring/alertmanager-smtp) and projects it into the alertmanager-smtp Secret (lines 13–15).
No SOPS-encrypted alertmanager secret exists in the repo (only top-level kubernetes/.sops.yaml and a migration guide).
The comment now accurately reflects the current architecture.
Recommendation
Looks reasonable — no blocking issues; this is a straightforward documentation fix.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `7e825aff0fe1acb3c6ab8b3dadd4ee6da3017d12`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR corrects a stale comment in the Alertmanager HelmRelease that incorrectly claimed the SMTP credential lives in a SOPS-encrypted Secret. The change is purely documentation—the comment now accurately points to OpenBao and the ExternalSecret that actually manages the credential. This fixes a real source of confusion for anyone debugging the `AlertmanagerClusterFailedToSendAlerts` alert, as the old comment would send them to a file that no longer exists.
## 🟢 Nits
None. The comment correction is accurate, minimal, and helpful. No YAML structure changes, no behavioral impact.
## Tests
Not applicable—this is a comment-only change. The rendered Alertmanager config will be byte-identical after Flux reconciles.
## Verification
Verified that:
- `alertmanager-smtp-externalsecret.yaml` correctly sources the credential from OpenBao (`apps/monitoring/alertmanager-smtp`) and projects it into the `alertmanager-smtp` Secret (lines 13–15).
- No SOPS-encrypted alertmanager secret exists in the repo (only top-level `kubernetes/.sops.yaml` and a migration guide).
- The comment now accurately reflects the current architecture.
## Recommendation
Looks reasonable — no blocking issues; this is a straightforward documentation fix.
<!-- pr-review:sha=7e825aff0fe1acb3c6ab8b3dadd4ee6da3017d12 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
This PR does not fix the firing alert — the alert is not repo-fixable.
Alertmanager*FailedToSendAlertsis firing because Proton's SMTP submission endpoint is rejecting Alertmanager's AUTH dialogue. The credential itself lives in OpenBao (apps/monitoring/alertmanager-smtp), not in this repository, so no manifest change can resolve it; the remediation is a human action described under Manual action required below. The only change here is a documentation correction: the Alertmanager comment block inhelmrelease-prometheus.yamlstill claimed the SMTP token lives in a "SOPS-encrypted Secret", which sends whoever responds to this exact incident to a file that does not exist. That comment is now corrected to point at OpenBao and the ExternalSecret.I am flagging two places where the issue's analysis did not match what I observed live this run — see Notes.
Changes
kubernetes/apps/monitoring/helmrelease-prometheus.yaml— corrected the stale comment aboveconfig:. The SMTP token is no longer in a SOPS-encrypted Secret; it is stored in OpenBao atapps/monitoring/alertmanager-smtpand projected into thealertmanager-smtpSecret byalertmanager-smtp-externalsecret.yaml. Comment-only; no rendered manifest change.Verification
Observed live this run (all read-only, via
./.ci-diag/kubectl):kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager→2/2 Running,0restarts, age5h26m. Alertmanager itself is healthy; this is not a pod/rollout problem.kubectl logs alertmanager-...-0 -n monitoring -c alertmanager --tail=30→ the error has changed since the issue was written. Up to23:28:04Zit is the reported 454; from23:28:14Zonward it is a different code:increase(alertmanager_notifications_total{integration="email"}[1h])→20.2;increase(alertmanager_notifications_failed_total{integration="email"}[1h])→other = 16.0, all other reasons (authError,rateLimited,serverError,clientError) =0. So roughly 4 of 20 sends in the last hour got through.kubectl get externalsecret -n monitoring alertmanager-smtp→Ready True SecretSynced,refreshTime=2026-08-26T23:34:25Z. Secret delivery into the cluster is working; this part of the issue's analysis holds.Repo-side verification of the comment fix:
kubernetes/apps/monitoring/alertmanager-smtp-externalsecret.yamlsourcesapps/monitoring/alertmanager-smtpfrom theopenbaoClusterSecretStore into a Secret namedalertmanager-smtp.find kubernetes -name '*sops*'returns onlykubernetes/.sops.yamlandkubernetes/MIGRATION-sops-openbao.md— there is no SOPS-encrypted alertmanager secret, confirming the old comment was stale.Reviewer check: this change is comment-only, so after Flux reconciles the rendered Alertmanager config must be byte-identical. Confirm the HelmRelease reconciles cleanly and that no Alertmanager restart or config reload is triggered. No alert will stop firing as a result of this PR.
Manual action required (not doable from the repo)
454 4.7.0 ... Connection lost to authentication server) is emitted when Proton's SMTP frontend cannot reach its own auth backend.535, regenerate the Proton SMTP token". That has now happened. However, the intermittent successes above are evidence against a fully revoked token: a rejected credential returns535on every attempt, and this one is being accepted several times an hour. Before rotating, re-check whether the535s have stopped, since rotation is disruptive and may be treating a provider-side symptom.apps/monitoring/alertmanager-smtp. ESO refreshes hourly (refreshInterval: 1h), so force a resync or wait for the next refresh.Notes
Two corrections to the issue, both grounded in output observed this run:
AlertmanagerFailedToSendAlertsnotification itself was delivered at23:30:38Z. Failures come in bursts: a 10-minute window ending ~23:44Zhad zero successes, while23:29–23:32Zhad four. This lowers the severity but does mean some notifications are still being dropped after retry exhaustion.authError = 0as proof the credential is fine. Alertmanager buckets SMTP authentication failures underreason="other", notreason="authError"— which matches exactly what I measured (other = 16,authError = 0) while the logs show nothing but auth errors. So that metric cannot rule a credential problem in or out. The real evidence that the credential is still valid is the intermittentNotify success, not the metric. The conclusion (provider-side fault) is still the best-supported reading, but not for the reason given.Deliberately not done:
max_retriesknob for email, and suppressingAlertmanager*FailedToSendAlertswould hide precisely the signal that outbound alerting is broken.Closes #649
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit7e825aff0fe1acb3c6ab8b3dadd4ee6da3017d12.Summary
This PR corrects a stale comment in the Alertmanager HelmRelease that incorrectly claimed the SMTP credential lives in a SOPS-encrypted Secret. The change is purely documentation—the comment now accurately points to OpenBao and the ExternalSecret that actually manages the credential. This fixes a real source of confusion for anyone debugging the
AlertmanagerClusterFailedToSendAlertsalert, as the old comment would send them to a file that no longer exists.🟢 Nits
None. The comment correction is accurate, minimal, and helpful. No YAML structure changes, no behavioral impact.
Tests
Not applicable—this is a comment-only change. The rendered Alertmanager config will be byte-identical after Flux reconciles.
Verification
Verified that:
alertmanager-smtp-externalsecret.yamlcorrectly sources the credential from OpenBao (apps/monitoring/alertmanager-smtp) and projects it into thealertmanager-smtpSecret (lines 13–15).kubernetes/.sops.yamland a migration guide).Recommendation
Looks reasonable — no blocking issues; this is a straightforward documentation fix.