Experimentation: putting research papers into prod
summary
Claims specific enough to act on: interleaving is two orders of magnitude more sensitive than classic A/B for ranking, but requires an abstraction library plus per‑item logging and performance guards; overlapping experiments let teams run more tests by partitioning product surfaces but add interaction effects and platform complexity; revising the metric hierarchy to favor feature‑level sensitive metrics and adding client instrumentation often cuts test duration from months to weeks.
AI-written summary · about
chapters (12)
- 0:00Experimentation and A/B testing
- 2:12Why AB testing is crucial in product development
- 5:43Why A/B testing skipped
- 9:15Rapid ML ranking iteration
- 13:32Interleaved comparisons in practice
- 17:03Insufficient user testing capacity
- 20:35Overlapping testing strategies
- 23:24Metrics not moving in tests
- 26:15Feature level metrics insights
- 28:02Solutions to A/B testing challenges
- 29:49Why innovate in experimentation
- 31:56Q&A
related talks
32mExperimentation: putting research papers into prod
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
28mExperimenting your way to a better product
36mA/B Testing in Production
48mHow to use experimentation in the back-end
28mOptimizing your A/B testing roadmap
35mExperimentation for Engineers
19mBeing Intentional: AB Testing and Data Minimization