An SLO-Driven Approach to Enhance Kubernetes Cluster Reliability
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Wednesday, 2020-11-18· at 17:45· Intrado Virtual Event Platform,·
summary
Ant Financial runs Kubernetes clusters with thousands of nodes and millions of pods and defines concrete SLOs such as 99% pods ready within 5 minutes measured end-to-end by correlating audit logs, events and pod names into a centralized log analysis that emits time-series. Component SLOs cover API server error/latency, reconcile error rate, queue depth and leader election. Monitoring uses Prometheus recording rules and an error‑budget alerting policy where 3% hourly burn triggers page and 5% daily burn files tickets.
AI-written summary · about
chapters (15)
- 0:00Thank You to Our Session Recording Sponsor
- 0:32Outline
- 1:07Motivation
- 1:52Fleet Management
- 3:48General Approach
- 5:07SLO Approach
- 6:52SLO Recap
- 7:48What SRE cares on K8S?
- 9:56EZE SLO Design
- 12:58Fine-grained SLO
- 13:53Component SLO
- 15:25Overall SLO Graph
- 16:37Why RatioRate is bad?
- 19:24Alerting Philosophy
- 23:32SLO Management
related talks
31mServerless Observability: a case study of SLOs
37mUsing SLOs for Continuous Performance Optimizations of Your K8s Workloads
27mScaling SLOs with Kubernetes and Cloud Native Observability
40mScaling SLOs with Kubernetes and Cloud Native Observability
30mOperating within Normal Parameters: Monitoring Kubernetes
39mKubernetes Scalability Definition Evolution
30mFrom Monitoring to Observability: Left Shift your SLOs with Chaos
33mHow and why we lowered our SLO