Optimal A-B Testing
Talkat Optimization and Decision-Making Under Uncertainty (Simons Institute, September 2016) · 2016-09-19..09-23
summary
Claims that optimizing allocation, not randomizing, can boost estimator efficiency from constant to order-n efficiency when covariate dimension scales with sample size; formulates offline problem with a provable approximation and an online control problem reduced to a fixed-state DP. Assumes known covariance and elliptical covariates, replaces empirical precision by its limit, then collapses state to imbalance and the Mahalanobis norm of covariate sum to get a practical near-optimal policy; shows gains on Yahoo clickstream and flags sparse/regularized models and post-hoc testing as open problems.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:12The Prototypical A-B Test
- 3:41A-B Tests Are Broadly Relevant...
- 6:32Statistical Efficiency is Important Practical Issue
- 8:03A Model Free Approach: Stratification
- 8:44Substantial Antecedent (+ Active) Literature
- 14:30A Common Treatment Model
- 16:29Estimating the Treatment Effect
- 18:21A Precise Problem Formulation
- 23:58The Offline Problem is Easy
- 25:23The Value of Optimization
- 26:17Randomization And (No) Learning
- 26:59Can Optimization Allow Learning?
- 28:35The Online Problem
- 30:19A Surrogate Problem
- 34:47The Surrogate Problem Yields Near Optimal Policies
- 36:52Ap-tim Dynamic Program
- 39:43Using The Insight
- 42:06State Space Collapse: From p-dim to 2-dim
- 43:26Example: The Impact On Web Impressions
- 43:45Open Problems