A/B Testing with SciPy
summary
Concrete recipe for running and evaluating A/B tests with SciPy: use scipy.stats.chi2_contingency or proportion z‑tests for categorical metrics, run the simulation notebook to see that 500 per arm only detects a 1% absolute lift about 74% of the time while ~6,500 per arm detects 5%→6% reliably, check assignment, logging and identity to avoid cross‑variant leaks, and use the Optimizely sample size calculator (or a multi‑armed‑bandit setup if you need adaptive allocation).
AI-written summary · about
chapters (8)
- 0:00Intro to A/B testing with SciPy
- 3:25What is A/B test
- 15:47Setting up A/B framework
- 27:50Analyzing A/B results with examples
- 34:28A/B test case studies
- 47:52Hypothesis testing in A/B
- 1:06:04Bayesian vs frequentist in A/B
- 1:09:30Q&A
related talks
30mStats Don't Have to Be Scary: Automatic a/b Test Analysis Using Python
01h 22mDoing frequentist statistics with Scipy
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
28mExpAn - A Python library for advanced statistical analysis of A/B tests
33mABBY: A Django app to document your A/B tests
01h 10mLies damned lies and statistics in Python
59mTesting in Python
32mExpAn: A Python library for advanced statistical analysis ...