Selection bias: The elephant in the room
summary
Problem: training labels come only from later funnel stages so measured gains don’t reflect true production performance; experiments with a family‑trip classifier show observed AUC can rise while actual AUC stays flat and variance concentrates on unlabeled regions. Key diagnostics: build a propensity model (predict s from X) to quantify exposure, compare observed vs actual performance, and measure prediction variance. Mitigations: more data reduces variance and bias, remove features that predict selection, or use inverse propensity weighting.
AI-written summary · about
chapters (8)
- 0:00Selection bias intro
- 1:23Family trip prediction case
- 13:57Measuring variance effects
- 18:12More data mitigates bias
- 23:10Propensity spectrum role
- 28:56Mitigation strategies trade-offs
- 30:32Summary call to action
- 31:15Q&A solutions
related talks
35mML: Beyond confusion matrix, model performance analysis
59mWhat Can Program Analysis Say About Data Bias?
43mThe Elephant in your Dataset: Addressing Bias in Machine Learning
53mAre We Measuring the Right Thing? Distribution Shift Lessons for Federated Learning
28mMeasuring Model Fairness
39mBeyond Bias: Crafting Fairness in Pythonic Machine Learning Systems
30mDetecting Hidden Bias in Datasets Before Models Fail
31mMeasuring Model Fairness