Check before You Change: Preventing Correlated Failures in Service Updates
summary
CloudCanary audits service-update snapshots in real time by reusing previous audit caches and translating dependency graphs into SAT problems, using SnapAudit for incremental checks and a min-cost SAT loop plus model counting in DepBooster to synthesize fixes. The system processes large deployments up to 200x faster than prior tools, generates improvement plans within minutes, and finds correlated failure risk groups with near-100% accuracy on production traces.
AI-written summary · about
chapters (17)
- 0:00Start
- 2:11State of the Art
- 3:15Proactive Auditing
- 5:18Correlated Failure Risks in Updates
- 7:09CloudCanary's Workflow
- 7:52Risk Groups in Fault Graphs
- 8:49The Insight of SnapAudit
- 9:43FirstAudit Primitive
- 12:22Original Deployment
- 12:51Find Border Nodes
- 13:58Identifying Risk Groups
- 15:36Merging Changed Caches
- 16:11Correlated Failure Risk Repairing
- 17:12Evaluation
- 17:50Efficiency Comparison
- 18:28Accuracy V.S. Efficiency
- 19:37Conclusion
related talks
21mContinuous Verification Updates
34mEnabling Invisible Infrastructure Upgrades with Automated Canary Analysis
35mReduce Risks With Continuous...
51mCutting edge Continuous Delivery: Automated Canary Analysis through Spring based Spinnaker
18mLimiting the Blast Radius: Preventing Outages from Bad Configs with Datacenter-Scale Health Checks
14mCanary Deployments for Infrastructure Upgrades
35mCutting-edge Continuous Delivery: Automated Canary Analysis on CF using Spinnaker
41mContinuous Well-Architected: Beyond the Once-a-Lifecycle Review with SCORP