A/B Testing in Production
summary
Starts with a concrete claim about experiment setup failures and shows numbers: a 1 000‑user baseline example, seeing 90 vs 128 conversions (≈0.028 lift) but insufficient confidence, alpha = 5% and 95% confidence interval rules; uses Evan Miller's a b testing calculator to show that a 20% baseline with 5% detectable effect needs ~1 030 sample users per variant. Covers tools (optimizely, Firebase, Facebook geolift, home‑built frameworks), AAA tests for randomization bugs, novelty effects, and overlapping experiment traps.
AI-written summary · about
chapters (13)
- 0:00A/B testing intro
- 1:33Measurable targets
- 5:26Conversion stats analysis
- 10:31Segmentation strategies
- 14:27Avoiding overlap effects
- 16:38Results with power
- 19:29Stop or revisit tests
- 20:34AA tests validation
- 22:40Q&A
- 29:06Peaking problem
- 30:52B2B user definition
- 32:38Avoid cherry-picking
- 34:07Small traffic & Bayesian
related talks
52mBuilding an A/B Testing Framework that works for Developers
27mCustomer Driven Development : What, Why & How?
41mA/B Testing with Style
28mOptimizing your A/B testing roadmap
46mAB Test Bloody Everything
34mA/B Testing You'll Like
36mHidden engineering challenges of A/B Testing
28mExperimenting your way to a better product