A Theory and Practice of Alerting with Service Level Objectives
summary
Argues for alerting on SLOs and error budgets instead of many cause-based pages, with concrete math: count requests inside the app or at the load balancer, compute SLI over a chosen window, extrapolate a burn rate from a short sample (e.g. 15 minutes) and page if that projection exhausts the budget within a chosen horizon (e.g. 24 hours). Demos a Prometheus + Go setup, shows replacing a backend raises the burn and triggers paging, and reports reducing pager load to under two pages per shift for four weeks.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:27Ops Overload
- 4:19What Is a Symptom Based Alert
- 10:27How Much Failure Is Acceptable
- 10:44Error Budget
- 26:04Alerting Threshold
- 27:14Cumulative Errors
- 31:09Observability
- 38:42Operational Excellence
related talks
43mSLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size
39mActionable Observability
36mHow to Include Latency in SLO-based Alerting
50mApplied Alerting Philosophy
33mTackling Alert Fatigue with SLOs
26mSLOwly Burning Out - Avoiding Common Pitfalls When Setting SLOs
38mImplementing SRE practices: SLI/SLO deep dive
19mLatency and Availability Error Budgets Done Right at Scale