Canarying Well: Lessons Learned from Canarying Large Populations
summary
Argues that canarying faces a concrete tradeoff between fast rollout, small blast radius, and metric coverage and gives rules to balance it: pick a handful of SLI-linked metrics, ensure canary and control are truly representative (watch regions, warm caches, bimodal replicas), stage with a short smoke canary then a longer sweep, and expect automation to break if many checks fail. Notes surprises: a 1% per-check failure rate across 100 checks yields ~2/3 releases needing human review, memory leaks demand long canaries, and Google’s Canary Analysis Service is used and published as a working approach.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:27Canarying: What is that?
- 2:34What we're going to talk about
- 3:26What we're not going to talk about
- 4:18Conflicting Incentives
- 5:47Triangle of Canarying Priorities
- 8:14Example: Geographical distribution
- 10:11Example: High variance among replicas
- 13:16Example: Bimodal distribution
- 13:44Example: Two metrics, different outliers
- 14:35Takeaways 2
- 18:30Example: Service With Cache, restarted
- 19:17Example: Memory leak canary
- 25:14Example: Before/after test
- 25:44Example Takeaway
- 27:02Example: Compound probability
- 29:40Beware Meta Analysis
- 30:59Prefer Few Metrics
- 32:30Canary In These 3 Simple Steps
- 33:59Canary In These 3-ish Simple Steps
related talks
21mI Don’t Always Test My Code, but When I Do It’s in Production
15mYahoo Case Study: Automated Canary Analysis using Spinnaker Kayenta
20mTo Go or Not To Go: 4 Error Metrics for Confident Release Validation
42mMetrics Driven Blue-green Deployments using Spinnaker’s Cloud...
06mMaintaining Performance and Reliability on the Edge
37mUniversal heap trimming with LeakCanary 3
29mTuning Argo Rollouts for Thousands of Workloads
27mA Safari of Kubernetes and its Natural Habitat