Uncertainty quantification: How much can you trust your machine learning model?
summary
Claims that conformal prediction gives provable coverage (pick alpha = 0.1 → 90% marginal coverage), that it is model‑agnostic and distribution‑free, and that it builds prediction sets from a calibration fold using a nonconformity score (example: 1 − model score on MNIST) are all stated and demonstrated. Tools shown include split conformal workflows and packages like Mappy (and a Julia implementation). Important caveats: calibration methods (e.g. temperature scaling) and the exchangeability assumption control set size and whether the coverage guarantee holds, and intervals may be wider than bootstrap.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:04Why uncertainty matters
- 3:59Humanizing AI with uncertainty
- 5:47Sources of uncertainty
- 7:10Measuring uncertainty
- 10:16Calibration techniques
- 12:25Conformal prediction
- 16:23Conformal prediction with MNIST
- 21:56Implementing conformal prediction
- 24:06Conclusion & Q&A
related talks
36mQuantifying uncertainty in Machine Learning predictions
35mUncertainty Unleashed
30mConformal Prediction: uncertainty quantification to humanise models
44mAccuracy Is Not Enough: Building Trustworthy AI with Conformal Prediction
37mQuantifying Uncertainty in Time Series Forecasting
22mConformal Inference with Tidymodels
44mHandling predictive uncertainty in Machine Learning
30mBoosting AI Reliability- Uncertainty Quantification with MAPIE