Scaling SLOs with Kubernetes and Cloud Native Observability
summary
Describes an implemented pipeline that turns SLOs into code: teams open a GitHub PR with an OpenSLO-like spec, automation generates Prometheus rules (using Sloth), and per-cluster Prometheus + Thanos provides global queries and long-term storage. Integrates 20 different observability backends (Elastic, New Relic, etc.) via an observability API, enforces error budgets in CI/CD to block deploys, and serves ~2,000 engineers. Focuses on pragmatic automation and onboarding, and reports that the system works in production. observability as code OpenSLO
AI-written summary · about
chapters (10)
- 0:00Introduction to SLOs
- 3:54SLA SLO SLI relationship
- 7:02Customer-centric SLO definition
- 10:57Challenges in scaling SLOs
- 13:03Observability as code approach
- 18:06SLO code with openSLO and sloth
- 20:32Scaling Prometheus with Thanos
- 22:43Multi-tool observability architecture
- 24:35Key lessons learned
- 25:57Conclusion and Q&A
related talks
37mUsing SLOs for Continuous Performance Optimizations of Your K8s Workloads
25mAn SLO-Driven Approach to Enhance Kubernetes Cluster Reliability
09mObservability in Kubernetes: A Signal- and SLI-Centered Approach to Reliability
16mShrinking the Observability Bill
24mTaming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...
30mOperating within Normal Parameters: Monitoring Kubernetes
22mHow to make Dev and Ops folks look good as an SRE
39mKubernetes Scalability Definition Evolution