A/B Testing in Production
summary
Covers concrete failures and fixes for running A/B tests in production: defines baseline conversion and detectable lift (example baseline 0.10 aiming for a 0.02 lift), simulates 1000 users per variant and shows a 0.028 observed lift that lacks confidence, uses Evan Miller calculator to compute required sizes (example 1030 sample users per variant for 20% baseline and 5% effect), warns about broken randomization, Optimizely Firebase vs home-built frameworks, and recommends segmentation, AA checks, and cautious stopping rules.
AI-written summary · about
chapters (24)
- 0:00A/B testing intro
- 1:12Measurable experiment targets
- 3:41Statistical confidence in results
- 5:46Central limit theorem & hypothesis testing
- 7:13Null vs alternative hypotheses
- 10:31Segmentation pitfalls in production
- 11:58Refining segments without bias
- 13:23Segmenting by customer value
- 14:27Focus on why not what
- 15:33Overlapping experiments & novelty effects
- 16:38Interpreting alpha and beta
- 18:23Sample size & detectable effect
- 19:29When to stop A/B test
- 20:34AAA experiments for validation
- 21:38Randomization & assignment bugs
- 23:00Q&A: Expected conversion increase
- 25:34Q&A: Group sizes & comparisons
- 27:39Q&A: Sample size per group
- 29:06Q&A: Peaking problem in early stopping
- 29:50Q&A: Incrementality testing & frameworks
- 30:52Q&A: Defining B2B user
- 32:38Q&A: Comparing multiple KPIs
- 34:07Q&A: Small traffic & Bayesian approach
- 35:45Closing remarks
related talks
46mAB Test Bloody Everything
52mBuilding an A/B Testing Framework that works for Developers
28mOptimizing your A/B testing roadmap
27mCustomer Driven Development : What, Why & How?
41mA/B Testing with Style
36mHidden engineering challenges of A/B Testing
34mA/B Testing You'll Like
39mDIY A/B Testing Architecture