Applying GitOps principles for central alert management
summary
Central alert storage in an Azure Git repo reconciles into Mimir Ruler via a git-sync sidecar, with folder-per-team containing Prometheus-style alert and recording YAML and Grafana JSON dashboards. PR pipeline runs validations (label enforcement, max rules per group, minimum eval/range intervals), syntax checks with promtool and lowkey linters, and optional unit tests that mock series; demo shows a dashboard rename and a new alert deployed. Current Mimir set is ~300 alerts, expected to grow to ~1–2k; limitation: no PV for git-sync causing startup failures if repo is unreachable.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 1:47Architecture and challenges
- 4:36GitOps solution design
- 9:20Validation and workflow
- 10:24Live demo and implementation
- 14:21Deep dive into validations
- 18:14Limitations and alternatives
- 21:17Q&A and concluding remarks
related talks
17mMonitoring as Code
48mAlert Management as Code
27mMonitoring as Code: Managing your Observability Assets the Cloud-Native Way
34mSneakOps – Getting Users To Use GitOps Without Them Even Knowing About It!
48mDevOps Transformation: Introducing Incident Management and Maximizing Monitoring Value
29mManaging infrastructure with Git: First principles of GitOps
46mAn IT Operations Approach to Observability
36m"Alert overload: How to adopt a microservices architecture..."