Experimenting your way to a better product
summary
Booking.com uses A/B testing as a core business process for over eight years, running hundreds of experiments where 9 out of 10 tests fail so teams favor tiny, focused changes and fast iteration. Tests use random assignment, a power calculator to size runtimes (round up to full weeks), and track meaningful outcomes not vanity metrics. Example experiments include adding a Butler filter or a map banner; negative results get budgeted learnings and drive new hypotheses.
AI-written summary · about
chapters (18)
- 0:00Introduction and personal story
- 1:48Why experiment in design
- 3:59A/B testing at Booking.com
- 5:44A/B testing process explained
- 8:57Formulating a user-centered hypothesis
- 10:00Creating and running the test
- 11:06Determining test duration
- 13:15Evaluating test results
- 16:24Meaningful metrics over clicks
- 17:07Learning from negative experiments
- 18:52Common A/B testing pitfalls
- 22:03Avoiding confirmation bias
- 23:07Statistical significance matters
- 24:11Testing with low traffic
- 24:53Validating surprising results
- 25:57Beyond A/B testing data
- 26:18Company culture for A/B testing
- 27:01Embracing failure and learning
related talks
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
28mOptimizing your A/B testing roadmap
52mBuilding an A/B Testing Framework that works for Developers
36mA/B Testing in Production
27mCustomer Driven Development : What, Why & How?
19mTurbo Charging your Product-Market Fit
35mExperimentation for Engineers