Discovering Your Model's Known Unknowns and Unknown Unknowns
summary
Active learning recipes with PyTorch examples show how to find a model’s known unknowns (least confidence, margin, ratio, entropy) and unknown unknowns (cluster sampling, model-based outliers, representative sampling), plus Monte Carlo Dropout for Bayesian uncertainty. Provides short runnable code, recommends calibrating neuron activations on held-out validation, and shows combining clustering with uncertainty prevents over-sampling one region. Claims softmax often misleads on confidence, MC Dropout is costlier but more reliable, and reports a striking 99% bias finding where contextual models prefer “his” over “hers.”
AI-written summary · about
chapters (5)
- 0:00Introduction and overview
- 1:05Known unknowns and unknown unknowns
- 3:57Uncertainty sampling methods
- 12:08Diversity sampling techniques
- 21:26Q&A and closing
related talks
46mKnow What You Don't Know: Tools to Understand Uncertainty in DL
35mUncertainty in deep learning
44mHandling predictive uncertainty in Machine Learning
35mUncertainty Unleashed
22mHow to Detect Silent Failures in ML Models
27mI have no clue what my model is doing...
37mActive Teaching, Human Learning
30mMachine Learning Techniques for Class Imbalances & Adversaries