A B Testing: Harder than just a color change
summary
Yelp’s Transaction Platform runs three experiments that together lift revenue 15 percent. A green-vs-orange button yields a 5.26% relative increase using a five percent minimum detectable effect and 95% power, so sample-size math gives about 300,000 users per variation. Implementation uses request bucketing and quarks, rolled out internally then 50/50 in production. A leftward layout boosts some clicks 10–25% but hurts ads so it doesn’t ship; concurrent testing uses swim lanes to avoid multiplicative variation explosion.
AI-written summary · about
chapters (22)
- 0:00A/B testing at Yelp
- 3:15What is an A/B experiment
- 4:20Running A/B tests step-by-step
- 5:24Case study: Button colors
- 10:17Control groups and bucketing
- 13:46Analyzing results and next steps
- 14:27Case study: Colors and layouts
- 17:59Swim lanes for complexity
- 22:15Second experiment results
- 23:58Trade-offs in layout changes
- 24:39Key takeaways and scaling
- 28:54Q&A: Success metrics
- 30:06Q&A: Team roles
- 31:58Q&A: Experiments vs vision
- 33:28Q&A: Technical details
- 34:16Q&A: Automation
- 35:41Q&A: Swim lanes advantages
- 37:07Q&A: Visitor vs customer bucketing
- 38:28Q&A: Experiment duration
- 39:43Q&A: Statistical significance
- 41:18Q&A: Prioritizing experiments
- 43:28Q&A: Planning and docs
related talks
52mAB-tests, lies, damned lies, and statistics.
41mA/B Testing with Style
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
34mMaximizing marketplace experimentation: switchback design for small samples
20mGrowth design as art and science
37mA/B testing at Zalando: concepts and tools
01h 15mAB-testing by cluster
19mPricing, Supply, A/B and Switchback Testing in FoodTech