Designing Online Advertising Markets
summary
Argues that standard user-level A/B tests systematically mislead advertisers because each sees only a tiny fraction of traffic, so short-term revenue can be misleading revenue; recommends measuring sustainable revenue, experimenting at advertiser or community level, and computing network-aware tests by selecting focal-buffer-auxiliary sets to turn non-sharp hypotheses into exact permutation tests. Develops causal forest and honest-tree methods that yield valid confidence intervals and asymptotic normality for heterogeneous treatment effects, and documents a concrete result: swapping positions cuts CTR by 13.6 percentage points.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:23Different Effects
- 5:52Theory
- 9:43AB Tests
- 14:12Cheaper fixes
- 16:23More expensive fixes
- 18:43Testing interactions
- 27:23Heterogeneous effects
- 31:28Regression tree
- 32:32Causal effects
- 34:11Heterogeneity
- 34:39Bias
- 35:24Random Forest
- 36:15Causal Forest
- 40:29Experimentation
related talks
28mA/B testing and canary releases - why, what and how?
48mHow to use experimentation in the back-end
52mDashboard Mechanisms for Online Marketplaces
41mTesting on Production, deep backend edition
56mMarket Design and Computer-Assisted Markets: An Economist's Perspective
29mClearing Matching Markets Efficiently: Informative Signals and Match Recommendations
44mAdvertising at the Speed of Light: DMP Platform
44mClearing Matching Markets Efficiently: Informative signals and Match Recommendations