The Challenge of Valid Evaluations
Talkat Bridging Prediction and Intervention Problems in Social Systems (Simons Institute, 2026) · 2026-01-12..01-18·
summary
Argues standard train/test evaluation misleads when decisions affect outcomes and measurements are proxies, and shows concrete methods to fix that: derive partially identified bounds under selective labels and one-sided measurement error, estimate bounds with doubly robust / double‑ML estimators built from efficient influence functions, and apply them to hospital datasets where covariate shift materially changes risk estimates; demonstrates bounds can cover true false‑positive/negative rates if the measurement parameter is plausibly bounded.
AI-written summary · about
chapters (9)
- 0:00Valid evaluations in algorithmic decisions
- 1:08Standard ML evaluation limits
- 3:23Selective labels in pneumonia risk
- 7:01Proxy outcomes in medical data
- 8:36Distribution shift in hospitals
- 10:05Validity threats in evaluations
- 15:20Statistical methods for validity
- 26:41Jointly handling shift and dependencies
- 29:14Summary and open challenges
related talks
35mEnsuring Reliable Evaluation Systems for Your ML
46mThe Measure and Mismeasure of Fairness
50mThe Measure and Mismeasure of Fairness
44mA Theoretical Framework for Statistical Evaluability of Generative Models
43mRobustness in High-Dimensional Inference Tasks
37mEfficient Algorithms for Reliable Machine Learning
23mHow to detect silent failures in ML models
09mTowards Credible and Effective Data-Driven Decision-Making