SLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size
summary
Argues that paging on user-facing symptoms and SLO error budgets cuts alert fatigue, shows a Prometheus implementation that measures SLIs at the load balancer, computes burn rate against a weekly SLO, and fires only when projected consumption threatens the budget. Demo runs a 10-backend round-robin app behind a load balancer, injects failures and triggers a page when the burn rate exceeds the 70 errors/sec threshold (roughly a 1% SLO baseline), proving the approach reduces noisy cause-based alerts.
AI-written summary · about
chapters (8)
- 0:00Start
- 8:12What Is Symptom Based Alerting
- 11:03Error Budgets
- 19:25Burn Rate
- 26:19Antagonistic Load Generator
- 30:33Firing Alerts in Prometheus
- 33:19How Do You Know What's Going Wrong in the System
- 39:46Acceptable Level of Paging
related talks
41mA Theory and Practice of Alerting with Service Level Objectives
26mSLOwly Burning Out - Avoiding Common Pitfalls When Setting SLOs
36mHow to Include Latency in SLO-based Alerting
33mTackling Alert Fatigue with SLOs
23mGrafana SLO Demo: Prioritize critical resources with SLO-driven IRM
39mActionable Observability
31mServerless Observability: a case study of SLOs
37mRelia...bility?