Quantifying uncertainty in machine learning models
summary
Random forest code uses fully expanded trees and keeps leaf training targets to build conditional distributions and 90% prediction intervals contain 91 of 100 truths on the superconductivity example; pruning or min_samples_leaf=600 collapses variance and coverage falls to 40 of 100. Quantile loss (fit with LightGBM or custom objective) produces percentile regressors; interval width correlates with error (Pearson ≈ 0.53). Classification probabilities need calibration via isotonic regression to become reliable.
AI-written summary · about
chapters (11)
- 0:00Trusting AI models
- 2:31Quantifying uncertainty matters
- 5:44Prediction intervals with random forests
- 10:03Evaluating interval accuracy
- 12:58Epistemic vs aleatory uncertainty
- 15:51Prediction variance from pruning
- 17:18Quantile regression for uncertainty
- 22:05Interval size vs error correlation
- 22:27Probability calibration in classification
- 27:49Solutions for calibration
- 29:17Key takeaways
related talks
34mPrediction intervals
31mUncertainty quantification: How much can you trust your machine learning model?
36mQuantifying uncertainty in Machine Learning predictions
33mAdaptive Prediction Intervals
35mUncertainty Unleashed
20mWhat's the uncertainty on your ML Prediction
37mQuantifying Uncertainty in Time Series Forecasting
44mHandling predictive uncertainty in Machine Learning