What's the uncertainty on your ML Prediction
summary
A Python library errortools implements uncertainty estimates by approximating the loss as a multivariate parabola to get a Gaussian parameter covariance, then propagating that to predictions either with linear error propagation or by sampling parameters. Examples use logistic regression on the Titanic and report an AUC 0.84 with confidence bands; per-example prediction intervals and metric uncertainty are shown. It works for models that minimize a loss, is on GitHub but not yet on PyPI, and tree models are currently unsupported.
AI-written summary · about
chapters (12)
- 0:00Uncertainty in ML predictions
- 1:23Defining uncertainty types
- 2:26Statistical uncertainty from data
- 4:11Estimating parameter uncertainty
- 6:40Likelihood for parameter uncertainty
- 9:06Visualizing parameter uncertainties
- 10:54Uncertainty propagation to predictions
- 13:24Visualizing prediction uncertainties
- 15:11Uncertainty for predictions
- 16:58Impact on model metrics
- 18:24Recap and future plans
- 19:48Next steps and questions
related talks
30mKnowing what you don’t know matters- Uncertainty-aware model rating
36mQuantifying uncertainty in Machine Learning predictions
33mQuantifying uncertainty in machine learning models
01h 24mBoosting AI Reliability: Uncertainty Quantification with MAPIE
12mOn the Use of ML for Blackbox System Performance Prediction
35mUncertainty Unleashed
30mBoosting AI Reliability- Uncertainty Quantification with MAPIE
22mHow to Detect Silent Failures in ML Models