Time, Interrupted: Measuring Intervention Effects with Interrupted Time-Series Analysis
summary
Explains how interrupted time‑series turns the counterfactual into a prediction problem, fitting Bayesian time‑series models (uses Prophet (Stan backend)) to pre‑campaign data and forecasting the period of a national TV buy. Shows posterior predictive samples used to get pointwise and cumulative prediction intervals, demonstrates a 10% simulated lift becoming statistically credible only cumulatively, and reports common failure modes: non‑stationarity, autocorrelation, confounding covariates, ramp‑up effects and model misspecification.
AI-written summary · about
chapters (31)
- 0:00Welcome!
- 0:46Introduction
- 2:41What is Interrupted Time Series Analysis
- 3:53A/B Testing
- 4:53How to measure the impact of a national TV campaign
- 6:25How can we know if something we did had an effect
- 8:05Counterfactuals
- 9:23Interrupted Time Series
- 13:01Building a time series counterfactual
- 14:01Non-stationarity
- 15:30Auto-correlation
- 16:15Independent and identically distributed assumptions
- 17:52What should the model include
- 19:45Prediction intervals
- 22:19Prophet library
- 23:26Training and prediction
- 24:53Assess accuracy of the model
- 26:20Compare predictions to observations
- 26:53Lift analysis
- 27:31Pointwise vs cumulative estimates
- 29:38Answering probability-based questions
- 30:05Threats to validity
- 30:53Change in the underlying process
- 32:49Confounding variables
- 33:47Model misspecification
- 36:27Q&A
- 38:26Situations where it worked or didn't
- 39:45Comparing different channels of advertisement
- 40:50Data preparation for Interrupted Time Series
- 41:55Ramp-up period before measuring the effect
- 43:06Assessing whether the counterfactual is correct
related talks
01h 30mTime Series Analysis with StatsModels
26mTime Series Analysis in Python: Easy Tools for Scientific Insight
17mThe causal problem of overexposure to repetitive ads: instrumental variables and pymc3
01h 12mIntroduction to Causal Inference
27mBenchmarking Time Series Foundation Models with sktime
33mAI-assisted coding, Python and dbt: taking Chronos live in two days
34mUsing Twitter's Breakout Detection Package
42mThe Duct Tape of Heroes Bayesian statistics