A/B Testing at Scale
summary
Booking.com runs experiments on "every pixel" across ~200 product teams and focuses on culture not just tooling, aiming for hundreds to thousands of tests a year. Success is roughly about 1 in 10 tests, and a "failure" means no insight rather than a negative metric. Scale comes from rules that require an experiment slot in roadmaps, pairing quantitative A/B results with qualitative research, and a self-sustaining peer-to-peer experiment review community plus ambassadors.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 3:33Failure as a learning opportunity
- 5:23Experimentation in decision-making
- 6:48Data and empathy balance
- 9:15Building experimentation culture
- 13:12Key takeaways summary
related talks
24mExperimentation Programs at Scale: Lessons Learned from Top Companies
24mDoes A/B Testing Work?
48mNo A/B — result XZ, or how we built an A/B testing platform at Ozon
36mExperimentation at Microsoft
50mCreating Performance Impact in a Changing Tech Landscape
40mReal-time A/B: how to get metrics for experiments
41mLittle-Known Pitfalls of A/B Testing
13mPersonalization and A/B Testing: How to create better experiences for users