The Data Addition Dilemma
summary
Combining datasets sometimes decreases target performance: experiments on the 208-hospital eICU corpus using an LSTM to predict 24-hour mortality show adding another hospital can raise or lower anchor-hospital AUC, with evaluations on the 12 largest hospitals producing both positive and negative deltas. A classifier-based divergence metric (scoreXY) correlates with AUC drop (p ≈ 1e-9) and selecting the top three sources by that score outperforms worst-three and simple mixture baselines; subgroup (racial) disparities respond inconsistently.
AI-written summary · about
chapters (5)
- 0:00Data addition challenges
- 5:50Healthcare data sources
- 13:19Costs and benefits of data
- 25:24Selecting data sources
- 36:52Q&A
related talks
53mAre We Measuring the Right Thing? Distribution Shift Lessons for Federated Learning
23mTransferring clinical prediction models across hospitals...
35mML: Beyond confusion matrix, model performance analysis
31mIndividual Gaps in Prediction Generalization
22mAI or Not AI? That's the Answer!
37mSelection bias: The elephant in the room
04mBlack Box: How AI will Amplify the Best and Worst of Humanity
24mHybrid ML Methods for the Interpretation and Integration of Heterogeneous Multimodal Data