Inference and Interference in Marketplace Experimentation
summary
Claims that marketplace A/B tests suffer both treatment-effect and standard-error bias, quantified in a continuous-time Markov chain model of a two-sided market; prior empirical biases range ~30% to 230% and simulation shows naive SEs understate true SE by about 30% for short experiments. Dynamics are regenerative so longer runs reduce SE bias but can increase false-positive launches; a calibrated time-series block bootstrap reduces SE bias and lowers false positives for positive interventions while naive longer horizons sometimes worsen decisions.
AI-written summary · about
chapters (8)
- 0:00Start
- 7:57Decision-making pipeline
- 11:37CTMC model of two-sided markets
- 13:55Running an experiment
- 18:05(2) Inference and SE estimation
- 29:29(3) Coverage of confidence intervals
- 30:17Implications for decision-making
- 32:27Fixing bias # improving decisions
related talks
39mCausal Effect Estimation in Practice: Lessons Learned from E-commerce & Banking
30mPolicy Evaluation under Interference
34mMaximizing marketplace experimentation: switchback design for small samples
33mTwo Years of Bayesian Bandits for E Commerce
41mKeynote: Personalization, Bandits, and Causal inference
27mA Practitioner’S Guide To Intent-To-Treat Effects From Multisite (Blocked)...
30mCausal Estimands And Interventions
33mHidden Biases In Data: Moving Beyond Black Box Models