1-5-10: How to Fast Recover Container Failure at Large Scale
summary
Claim: Alibaba runs millions of containers and targets 1-5-10 (1m/5m/10m) detection-to-resolution. Implementation uses a lightweight host agent, a Prometheus adapter plus other monitoring backends, and a matching rule engine that maps metrics to recovery actions in a container debugger; simple network/node faults auto-heal and over half of recoveries run remotely, while complex cases trigger human follow-up. Simulated rollouts and incident postmortems show many incidents miss the 1-5-10 targets, prompting progressive rollout, canary releases and automated remediation.
AI-written summary · about
chapters (14)
- 0:00Start
- 1:03Agenda
- 1:55Definition of 1-5-10
- 3:38Incident Timeline
- 6:59What we learned
- 8:34How to improve
- 11:08Postmortem
- 14:17Failure Prevention
- 14:57Chaos Engineering
- 15:37Progressive Deployment
- 18:18Observability
- 22:59Auto Healing
- 25:01Container Debugger
- 26:32Prophet
related talks
30mWhat Happens When Containers Fail
32mIncreasing your applications' availability with Amazon CloudWatch
37mAutomatically Detect the Top Performance & Scalability Issues in Distributed
29mProduction Cluster Monitoring and Remediation for High Reliability
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
34mWhat Happens When Something Goes Wrong? On Kubernetes Reliability
42mMaking Resilient, Self-Adaptive, And Autonomous Kubernetes Clusters
20mUsing Monitoring to Speed Up Your DevOps Journey