How to A/B test with confidence
summary
Covers how to run valid A/B tests: use a proper randomizer (not user-id parity assignment), compute sample size from test power and choose the right statistical test for the metric, and avoid peeking or repeated significance checks. Gives concrete failures: a variant that caused 10× more crashes (contaminating results), a nonrandom split that hid pre-test revenue differences, and a single Bill Gates outlier donation that wrecked a revenue test. Recommends tools like Vanity/Split, AA tests for rehearsal, and documenting scope, metric and outlier policy.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:06What is an A/B test
- 4:35What can go wrong
- 10:16Test setup flaws
- 13:44During the test
- 15:08Analyzing the results
- 20:46A/B tests at Recipe
- 23:14Best practices
- 27:08Q&A
related talks
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
34mA/B Testing You'll Like
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
24mThe Scientific Method of Experimentation
29mIntuitive A/B Test Evaluations for Coders
09mA/B tests: Understanding user behaviour