Balancing Personalization & A/B Testing in a Customized World
summary
Compares classic A/B testing (example: Booking.com changed “Book now” to “I'll reserve” and measures conversion rates) with adaptive approaches that reallocate traffic using multi-armed bandit methods, explains real-time Bayesian updates via Thompson sampling, and quantifies harm with a regret metric that measures lost engagement during exploration. Covers segmentation strategies, parametric bandits for performance marketing, sample-size and attribution-window limits, and when to prefer standardization versus personalized, adaptive experiments.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:31A/B testing basics
- 3:39A/B testing at Booking.com
- 7:12Personalization challenges
- 10:29Balancing personalization & testing
- 11:32Limitations of A/B testing
- 12:55Multi-armed bandits
- 17:26Bandits reduce user regret
- 20:57Key takeaways
- 21:38Q&A
related talks
21mLeveraging Multi-Armed Bandit Algorithms for Dynamic Decision Making
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
27mHow to A/B test with confidence
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
34mA/B Testing You'll Like
32mMulti-Armed Bandits in Recommendations