Building Causal Measurement
summary
Twitter builds an in-house incremental measurement system that layers incremental holdouts on top of standard A/B tests: each holdout runs 12 months split into a 6-month warm-up and 6-month reporting window, with teams keeping two overlapping cohorts. Implementation requires maintaining parallel code paths and nontrivial engineering work, but it surfaces cumulative impact, validates short-term novelty effects, reveals compounding or cannibalization between features, and supports detecting tiny lifts (fractions of a percent) that matter at scale.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:08Talk inspiration
- 3:33Why standard experiments fail
- 7:47Short-term A/B limits
- 10:17Incremental holdouts for measurement
- 12:06How holdouts work
- 16:40Lessons from holdouts
- 22:28When a holdout is not yet necessary
- 24:35Holdouts in measurement strategy
- 26:42Final reflections
- 28:26Q&A
related talks
46mFrom correlation to causation: measuring ad effectiveness at scale
47mIncrementality Bidding and Attribution
26mThe Need for People Based Measurement
28mCorrelation Sucks - Long Live Causality! Mastering Causal Inference in Data Science
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
27mHow to A/B test with confidence
38mContinuous Marketing Effectiveness Measurement
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)