Distribution-Free, Risk-Controlling Prediction Sets
summary
Proposes distribution-free procedures that wrap any black-box model to produce risk-controlling set-valued predictions: construct a one-dimensional family T_lambda whose risk is monotone, estimate a pointwise upper confidence bound for risk from a held-out calibration set, and choose the smallest lambda with bound ≤ gamma to guarantee P(risk > gamma) ≤ delta. Works for classification, segmentation, multi-label and more, gives finite-sample guarantees and needs a calibration set ~500–1000.
AI-written summary · about
chapters (5)
related talks
31mUncertainty quantification: How much can you trust your machine learning model?
36mQuantifying uncertainty in Machine Learning predictions
35mUncertainty Unleashed
33mOnline Multivalid Learning: Means, Moments, and Prediction Intervals
53mDesigning Robust Learners
25mConfidence-Based Active Learning
44mHandling predictive uncertainty in Machine Learning
47mA Multi-Group Perspective on Fairness