Knowing what you don’t know matters- Uncertainty-aware model rating
summary
Claims a concrete uncertainty-aware model rating that scores probabilistic forecasts against an ideal Poisson baseline using Python tooling and M5 data. Method buckets predictions logarithmically, fits dispersion scaling (empirically ≈ exponent 1.5), and yields numerical ratings that separate models: postdiction ≈100, naive ≈41, simple 28-day OK, LightGBM good. It builds a ratings pipeline, shows where improvements actually matter across departments and prediction ranges, and demonstrably distinguishes models and datasets.
AI-written summary · about
chapters (8)
- 0:00Uncertainty-aware model rating intro
- 2:52From description to interpretable ratings
- 5:41Perfect model evaluation procedure
- 9:14Bucketing predictions by velocity
- 12:43Scaling ratings across ranges
- 16:17M5 competition real models
- 21:12When to use uncertainty ratings
- 23:39Q&A
related talks
30mMy forecast is better than yours! What does that even mean?
20mWhat's the uncertainty on your ML Prediction
43mProbabilistic Programming in the Real World
31mUncertainty quantification: How much can you trust your machine learning model?
44mHandling predictive uncertainty in Machine Learning
37mQuantifying Uncertainty in Time Series Forecasting
36mQuantifying uncertainty in Machine Learning predictions
33mQuantifying uncertainty in machine learning models