Going from 30 to 30 Million SLOs
summary
Describes engineering work that scales SLOs from ~30 to roughly 30 million by moving from zone-level to per-customer SLO budgets, building a daily analytics pipeline that materializes SLO definitions from version control and runs in Dremel/BigQuery, and replacing histogram percentiles with a count-based latency SLI. Introduces the rule of five errors to avoid false positives on low-QPS customers, surfaces affected projects in dashboards, and shows that the system significantly improves detection and prioritization of real customer impact.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:38Control plane overview
- 2:18GCE resources and scale
- 3:40SLI, SLO, and SLA concepts
- 5:25Measuring latency SLOs
- 8:34Evolution of GCE SLOs
- 10:32Scaling and customer impact
- 13:00The curse of dimensionality
- 15:34Quantifying affected users
- 18:30Data pipelines and tools
- 20:26Dashboard reporting and fix
related talks
30mTechniques for SLOs and Error Budgets at Scale
17mRolling out Service Levels across a large org
32m9 Things You Should Do When Starting to Use SLOs
33mHow and why we lowered our SLO
37mReal World SLOs and SLIs: A Deep Dive
22mLook Ma, No Metrics: A Quest Towards Extracting SLOs
40mScaling SLOs with Kubernetes and Cloud Native Observability
41mSLOs that Delight: Master Multi-Service SLO Calculations