Little-Known Pitfalls of A/B Testing
summary
Explains common hidden pitfalls in running A/B tests, naming assumptions behind popular automation tools such as true random user assignment and independence of measured events. Describes why those assumptions often fail in real systems and what to watch for when designing and interpreting experiments.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 7:09Need for control experiments
- 11:25User split mistakes
- 18:11Production and data center issues
- 22:34Dependent measurements and analysis errors
- 31:45Offline testing and calculation errors
- 33:08Monitoring in experiments
- 38:48Q&A
