Enabling Invisible Infrastructure Upgrades with Automated Canary Analysis
summary
Pinterest describes a production canary pipeline that upgrades OS, runtimes and container images by comparing a control cluster to a canary and running statistical checks (Mann‑Whitney U) against metrics in OpenTSDB. It integrates Spinnaker + Kayenta, provisions canaries via EC2 and Teletraan, and uses a 700-line Python CLI plus an OpenTSDB plugin. Implementation takes ~1 year from 2–3 engineers, requires ~50 data points, adds 30–120 minutes per deploy, and reportedly detected >50% of issues before full rollout.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 2:02Target Audience
- 4:51Invisible and Infrastructure
- 6:11Defining Infrastructure
- 7:43Canary Analysis
- 8:54Infrastructure Expiration Date
- 9:47Business Needs
- 11:02Upgrading is Hard
- 11:50Automation
- 12:41Canary Analysis is not magic
- 14:11The typical migration
- 18:12What you need
- 19:11Example system
- 20:29Best practices
- 22:02Overall effort
- 22:41Lessons learned
- 23:46Additional resources
- 24:14Pinterest
- 24:29SRT Con
- 25:44UX
- 26:50Statistical Analysis
- 29:07Stateless App
- 29:37Solutions
- 30:45How long does it take
- 31:31How does it work
- 32:15False positives
related talks
51mCutting edge Continuous Delivery: Automated Canary Analysis through Spring based Spinnaker
43mCanary Deploys with Istio: Lessons Learned
14mCanary Deployments for Infrastructure Upgrades
35mReduce Risks With Continuous...
35mCutting-edge Continuous Delivery: Automated Canary Analysis on CF using Spinnaker
15mYahoo Case Study: Automated Canary Analysis using Spinnaker Kayenta
40mHow We Introduced CI/CD For...
10mLightning Talk: Open Source Canary Deployments with Application Metric Analysis