The Truth About Your Lying Calibrated Forecaster: How to Design Truthful Calibration Measures
summary
Shows that standard calibration measures like ECE are gameable: a manipulative forecaster gets ECE≈0 while a truthful forecaster pays Θ(√T) variance or even ω(T) in tailored examples. Introduces subsampled smooth calibration error (SSCE), combining Lipschitz smoothness and random subsampling, proves SSCE is sound and complete and multiplicatively truthful (constant-factor optimal, no additive gap), relates guarantees to a conditional-variance complexity σ, and gives algorithms achieving o(T) SSCE with practical computation via flow solvers and sampling.
AI-written summary · about
chapters (8)
- 0:00Calibrated forecasters intro
- 1:05Event forecasting and calibration
- 7:27Calibration measures
- 13:13Untruthful calibrated forecasters
- 21:03Truthful calibration measures
- 25:28Smooth calibration error
- 31:25Summary and implications
- 32:49Q&A
related talks
36mQuantifying uncertainty in Machine Learning predictions
01h 21mMulticalibration: Towards Fair Decision Making
47mA Multi-Group Perspective on Fairness
31mUncertainty quantification: How much can you trust your machine learning model?
45mMulticalibration, Universal Adaptability and Causality
37mOmnipredictors
38mTracking and Improving Information in the Service of Fairness
48mOnline Adversarial Multicalibration And (Multi)Calibeating