Alertmanager on Its Way to High Availability
summary
Alertmanager adds a high availability mode that gossips only metadata (silences and sent-notification records) between instances using gossip protocol and CRDTs (conflict-free replicated data types) over the mesh library, keeping alerts themselves out of the gossip. It generates deterministic notification IDs by hashing group labels plus route and receiver, implements at-least-once delivery with dedup windows (group_wait/group_interval), and demos three instances plus a webhook where only one JSON notification is emitted, demonstrating the approach works.
AI-written summary · about
chapters (16)
- 0:00Start
- 1:02Where does Coreos fit in?
- 1:29Alertmanager Features
- 2:54Prometheus & Alertmanager
- 4:08Grouped in one notification
- 4:58Infrastructure Scaling Story
- 8:43Example Alerting Rule
- 9:44Alert Evaluation in Prometheus
- 10:12Simple configuration
- 11:08Notification Pipeline
- 12:18What is gossiped?
- 13:46How? mesh by Weaveworks
- 15:40Create Silences
- 17:52Non silenced alert example
- 19:23Gossip Partition
- 20:25Notification Log
related talks
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
39mAlertmanager Has Amnesia – Should We Fix It?
24monotify: A scalable, flexible Alertmanager
07mLightning Talk: Alert Quorum Universal Aggregator - AQUA
43mMulti-Cluster Alerting: A Kubernetes-Native Approach
04mLightning Talk: Managing Silences - Provisioning Silences into Alertmanager
50mApplied Alerting Philosophy
48mAlert Management as Code