Accounting for user behavior data in A/B testing metrics
summary
Covers how to incorporate user behavior data into A/B testing metrics to improve sensitivity when sample sizes are limited. Describes problems with detecting winners for low-signal metrics (like loyalty) and discusses using behavioral data to make metrics more informative for online experiments.
AI-written summary · about
chapters (10)
- 0:00A/B testing introduction
- 3:59Why A/B tests aren't simple
- 7:30Stratification in experiments
- 10:42Delta-based metrics
- 14:59Validating the new approach
- 20:46Metrics pitfalls
- 24:57Handling new users
- 29:54Experiment duration trade-offs
- 35:32Modern techniques and ML
- 39:55Q&A
related talks
40mReal-time A/B: how to get metrics for experiments
43mA/B Testing: What to Pay Attention To
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
30mMeasurement in a Privacy First World
12mDid It Work? Measuring the Impact of Product Launches
18mSession: The Science Of Success – Driving Growth With Prediction, Incrementality, And Scalable....
28mMeasurement and Analysis for Sustainable Service Delivery
34mA\B Test Best Practices and Fun Case Studies from Deliveroo