A/B testing and canary releases - why, what and how?
summary
Gives concrete lessons: calculates required sample size for a 10%→12% lift, shows how sequential testing and a Bayesian (71% probability example) flow shortens experiments, and names tools like VWO and Optimize. Covers real failures (boiled orange analogue, 3D Secure rollout that broke metrics until session replays found the bug), clustering to avoid network interference, and when to restart a variant after changing it — worth watching if wanting actionable how-to and pitfalls.
AI-written summary · about
chapters (11)
- 0:00Start
- 3:461. Don't trust your gut feeling, Test your changes.
- 12:55Defining sample size A
- 14:07Peeking problem..
- 17:28Insufficient traffic
- 18:44Bayesian approach
- 20:08Something's off? Learn why
- 21:53Modifying variants?
- 22:29Interference
- 24:00Sample ratio mismatch
- 27:05The DETOUR act proposal
related talks
41mTesting on Production, deep backend edition
15mWhy I Prefer Good Testing Over Excellent Testing
42mTesting in production for fun and profit
46mA Practical Guide to Testing in DevOps
53mObservability Driven Development
48mHow to use experimentation in the back-end
49mDesigning Online Advertising Markets
32mI Test In Production