Alertmanager Has Amnesia – Should We Fix It?
summary
Alertmanager loses in-memory alert store on reboot by design, but DNS drops and clustered replicas cause missed inhibitions and duplicate notifications. Using Thanos/Prometheus as rulers, the team prototypes a shared persisted view — first a hackathon ETCD idea then a pragmatic on-disk snapshot restore written at shutdown/read at startup — and measures about an 80% drop in quarantine events and much quieter rollouts, though the patch needs wider validation and isn’t upstream-ready.
AI-written summary · about
chapters (8)
- 0:00Alertmanager challenges
- 5:20Alertmanager internals
- 10:55Noisy Tuesday incidents
- 15:12HA Alertmanager limits
- 20:23Testing Alertmanager fix
- 24:55Patched rollout results
- 26:00Fix Alertmanager amnesia?
- 27:44Q&A
related talks
32mAlertmanager on Its Way to High Availability
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
36mUnderstanding and Extending Prometheus AlertManager
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
18mAlert Analytics
38mEliminating Alerts or “Operation Forest”
33mAlerting in Cloud Native Environments
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise