Lightning Talk: Managing Silences - Provisioning Silences into Alertmanager
summary
Company runs HA Alertmanager per Kubernetes cluster and manual silences across many clusters proves unscalable and error prone (engineers set expiries to year 4000 or omit context). Built a silence operator (CRD) that stores silences in Git alongside Prometheus rules, validates expiry and metadata in CI, and uses the Alertmanager Go client to sync and auto-remove expired silences. Operator supports Mimir tenancy and per-cluster reconciliation, and is deployed in production for automated, auditable silence management.
AI-written summary · about
related talks
32mAlertmanager on Its Way to High Availability
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
18mAlert Analytics
05mLightning Talk: How the Observability Team at Spotify Radically Decreased On-Call Alerts
43mMulti-Cluster Alerting: A Kubernetes-Native Approach
19mSensory Friendly Monitoring
36mConfidence with Chaos for Your Kubernetes Observability
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise