Testing, Testing: On Experimental Drift and Data Driven Product Design
summary
Claims that A/B tests break when the environment shifts: a measured 5% lift from a February experiment on pet beds flips by summer because the user mix moves from a 50/50 cat/dog split to more dog buyers. Platform detail: a homebuilt Shopify A/B system with Bayesian defaults and assignments. Practical mitigations: monitor feature distributions (Kolmogorov–Smirnov), model feature-shift or retrain selectively, consider bandits, and keep a small eternal control group (~5% of new users) to preserve a stable baseline.
AI-written summary · about
chapters (14)
- 0:00A/B testing introduction
- 3:16Why A/B testing works
- 8:11A/B testing pitfalls example
- 11:19A/B test reliability assumptions
- 14:06Product flux curse
- 16:12How A/B testing problems affect machine learning
- 18:16Monitoring experimental drift
- 21:26Eternal control group
- 23:13Testing as limited resource
- 25:58Q&A: Platforms and AI
- 31:21Q&A: Control group approach
- 34:11Q&A: Multiple product versions
- 35:57Q&A: Incremental vs visionary
- 37:19Closing remarks
related talks
25mMake a Difference with Simple A/B Testing
33mHow to A/B test with confidence
09mDemystifying the Dreaded A/B Test
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
18mData Science Careers via A/B Testing
28mOptimizing your A/B testing roadmap
32mAB Testing to Determine Product Priority
24mDoes A/B Testing Work?