Testing in production for fun and profit
summary
Argues that testing in production is inevitable and shows concrete techniques used at Wikimedia to make it safe and measurable: canary rollouts, infrastructure-level A/B testing driven by a client-side random id stored in localStorage and cookies, separated caches, dashboards and clear acceptance criteria. Describes migrating from hhvm to php7 over six months, ramping anonymous traffic to 50%, requiring metrics within ~5% of hhvm, and completing the move with no user-visible outage.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:35Welcome
- 2:00Goal
- 2:40Overview
- 4:35Show Hands
- 5:20Testing in production
- 6:10Embrace it
- 6:40The staging problem
- 8:30Example
- 9:40Bug
- 13:50Scannering
- 18:00Phase rollouts
- 19:10Phase rollouts example
- 22:10Infrastructure labels testing
- 24:35Functionality
- 27:25Performance
- 28:30Metrics
- 29:05Testing
- 35:45How it went
- 36:20Conclusion
