Bayesian Deep learning with 10% of the weights
summary
Claims a variational Bayesian net that learns Gaussian mu and sigma per weight and samples weights at inference to get a posterior predictive, implemented in pure TensorFlow. Empirical result: networks trained with this objective can prune 90% of parameters (often keep performance down to ~10% or less, even on a 500‑sample UCR EKG with ~500k weights) and produce useful uncertainty measured as entropy of the predictive distribution, correlating with noise, warping, rotation and helping resist adversarial failures.
AI-written summary · about
chapters (17)
- 0:00Bayesian deep learning intro
- 1:03Uncertainty and parameter count
- 3:52Bayes rule and variational inference
- 6:21Approximating parameter distributions
- 9:49Measuring uncertainty with entropy
- 11:17Pruning with Bayesian inference
- 13:11Results on MNIST and ECG
- 20:01Live demo with drawings
- 24:40Key takeaways
- 26:05Q&A pruning and retraining
- 27:32Q&A continued training after pruning
- 28:17Q&A feature insights from pruning
- 29:04Q&A variational inference implementation
- 30:07Q&A prediction with distributions
- 31:11Q&A ECG performance dip
- 32:42Q&A preventing overfitting
- 34:06Q&A entropy for uncertainty
related talks
35mUncertainty in deep learning
39mBayesian Deep Learning with Edward (and a trick using Dropout)
12mBayesian Neural Networks With Variational Inference in PyTorch
45mStudying Generalization in Deep Learning via PAC-Bayes
34mBayesian Optimization and it's application to Neural Networks
44mAre You Sure about That?! Uncertainty Quantification in AI
01h 02mTutorial on Deep Learning I
25mHyperparameter Tuning and Visualization in Deep Learning