Always Valid Inference: Continuous Monitoring of A/B Tests
Talkat Optimization and Decision-Making Under Uncertainty (Simons Institute, September 2016) · 2016-09-19..09-23·
summary
Optimizely implements always-valid p-values computed from a MSPRT (mixture SPRT) so type I error remains bounded under arbitrary user stopping rules (fixes the 30–40% false positives seen with classic p‑values under continuous monitoring). The MSPRT gives power‑one, is tunable by matching mixing variance to an empirical prior of effect sizes, runs in-browser for real‑time updates, and is deployed since Jan 2015 across thousands of customers.
AI-written summary · about
chapters (16)
- 0:00Introduction to AB testing
- 0:36Classical hypothesis testing
- 3:15The problem with P values
- 6:58Risks of continuous monitoring
- 9:41Modeling user behavior
- 12:30Outline of proposed solutions
- 15:38Always valid P values
- 18:53Implementing the new approach
- 21:53User-centric trade-offs
- 24:27The msprt test explained
- 29:25Efficiency and optimality
- 33:27Risk preferences in practice
- 35:42Cost-benefit and prior models
- 38:30Optimizing mixing distributions
- 42:00Performance and comparisons
- 47:29Conclusion and Q&A
related talks
42mA/B testing with real-world data to evaluate ML model updates
19mHow Etsy Handles “Peeking” in A B Testing
34mHypothesis to Impact: Mastering the Art and Science of Controlled Experiments
33mHow to A/B test with confidence
41mMutation testing in practice
28mOptimizing your A/B testing roadmap
38mTesting, Testing: On Experimental Drift and Data Driven Product Design
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%