expectation: A modern take on statistical A/B testing with e-values and martingales
summary
A new Python library called expectation implements e-values and martingales for sequential A/B testing, change‑point detection and confidence sequences, with demos showing a simulated 10%→12% uplift detected within a few days and early stopping that remains valid. The framework treats accumulated evidence as a wealth process starting at $1, ties rejection thresholds to money (e.g. $20 ≈ 1/0.05, $100 ≈ 1/0.01), relies on Ville's inequality for type‑I guarantees, and ships as modular, object‑oriented, research‑driven code.
AI-written summary · about
chapters (13)
- 0:00Expectation library intro
- 1:55Why create expectation
- 3:39E-values intuition
- 8:55Sequential testing importance
- 10:45E-values definition
- 14:57Martingale wealth analogy
- 19:52Demo sequential A/B testing
- 25:53Early stopping rules
- 30:59Library features
- 33:27Change point detection
- 35:38Demo conformal E-testing
- 38:51Key takeaways
- 40:08Q&A
related talks
38mUsable A/B testing – A Bayesian approach
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
25mScaling Bayesian Experimentation
57mSearch Relevance: A/B testing to Reinforcement Learning
01h 10mBayesian Optimization and Machine Learning for Accelerating Experiments in the Physical Sciences
29mA Pre-Expectation Calculus for Probabilistic Sensitivity
33mSequential Testing Simplified with Basemath
39mSuccessfully Applying Bayesian Statistics to A/B Testing in Your Business