Are We Measuring the Right Thing? Distribution Shift Lessons for Federated Learning
summary
Large deployed sepsis model that reported AUCs ~0.76–0.83 falls to ~0.63 at one hospital, alerts on ~18% of admissions and only adds ~7% more sepsis detections at huge false‑positive cost, motivating a taxonomy: a) interpolation regimes where pooled ERM under a convex‑hull/coverage assumption works and shows accuracy‑on‑the‑line; b) adaptation regimes where fine‑tuning or last‑layer retraining helps; c) hard out‑of‑distribution cases needing causal/invariance assumptions. Common benchmarks using simple Dirichlet partitions often mask real heterogeneity.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:21Motivation for distribution shift
- 3:39Sepsis model failure case study
- 10:24Non-IID challenges in FL
- 13:10Three regimes of heterogeneity
- 18:55Accuracy-on-the-line explained
- 24:45ERM baseline in FL
- 27:22Adaptation strategies for shifts
- 31:16Global vs local objectives
- 33:34Pareto frontier in personalized FL
- 40:41OOD generalization challenges
- 44:05Benchmarking construct validity
- 48:38Summary and call to action
- 50:27Q&A
related talks
37mThe Data Addition Dilemma
35mML: Beyond confusion matrix, model performance analysis
23mTransferring clinical prediction models across hospitals...
31mIndividual Gaps in Prediction Generalization
45mManaged federated learning on AWS: A case study for healthcare
37mSelection bias: The elephant in the room
41mDistribution shift in ecological data: generalization vs. specialization,
22mAI or Not AI? That's the Answer!