A/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
summary
Booking.com has run AB testing for 8 years, building its own platform to run millions of experiments and detect small effects. A green vs blue button test shows 8.3% clicks on both variants but lower bookings from the green version, so the original stays. Common failures include big‑bang redesigns that hide causal signals, testing only landing pages, and assuming reproducibility across products; reliable practice is to test atomically, require sufficient traffic and clear business metrics, and treat tests as a learning corpus.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:45About Stuart
- 2:00Why AB Testing
- 3:30Design Phase
- 4:17Experimentation
- 4:55Why should you do experiments
- 7:40Common mistakes
- 10:15Presentation
- 10:26Hamburger Menu
- 10:54Context
- 11:54What does AB testing look like
- 12:13Test everything
related talks
28mExperimenting your way to a better product
34mA/B Testing You'll Like
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
46mAB Test Bloody Everything
27mHow to A/B test with confidence
29mBalancing Personalization & A/B Testing in a Customized World
24mThe Scientific Method of Experimentation
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)