Watching the Watchers: Generating Absent Alerts for Prometheus
summary
Prometheus rules parser extracts every selector from alert and recording rules, auto-generates a matching absent alerts rule by wrapping selectors in absent_over_time or absent() and emits an automatically maintained absent.rules.yaml file; CI commits these changes on PRs. Team replaces some instant vectors with 15m range vectors, initializes key counters to zero, and finds dozens of missing CloudWatch-exported metrics; tool runs eight months with no production runtime service and materially reduces silent failures.
AI-written summary · about
chapters (4)
- 0:00Start
- 2:46The manual approach
- 3:51The anatomy of a Prometheus expression
- 4:50Prometheus alert rules
related talks
17mMonitoring as Code
33mAlerting in Cloud Native Environments
50mMetrics and alerting with Prometheus and Grafana
18mAlert Analytics
02mWhy you use Prometheus for monitoring?
29mWho Watches the Watchers? From Closed Observability To...
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
08mPrometheus in 5 minutes