Studying Generalization in Deep Learning via PAC-Bayes
summary
Work-in-progress applies PAC-Bayes to predict SGD generalization by building Gaussian fractional-data prior learned on a subset S' (sometimes via differential privacy) and a Gaussian posterior centered at SGD weights; experiments on MNIST/Fashion/CIFAR show using a small S' reduces the KL penalty and tightens bounds, an oracle prior variance would help a lot, while a fresh “ghost” sample hardly improves results. Conclusion: randomized classifiers still underfit, KL term dominates, and non-vacuous bounds for deterministic SGD remain elusive.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:40What might generalization theory offer deep learning?
- 3:24Barriers to explaining generalization
- 4:20PAC-Bayes yields risk bounds for Gibbs classifiers
- 5:16PAC-Bayes generalization bounds
- 7:47PAC-Bayes bounds on deterministic classifiers
- 15:37Distribution-dependent approximations of optimal priors via privacy
- 19:32A question of interpretation
- 24:38Use SGD to predict SGD
- 28:51Data and distribution priors for neural networks
- 29:54MNIST Results - Coupled data dependent priors and posteriors
- 31:43Oracle access to optimal prior covariance
- 32:58Bounds with oracle covariance + ghost sample
- 33:34Bounds on 32k samples v 64k samples
- 34:07Recap and Conclusion
related talks
01h 07mLarge-Scale Private Learning, Part II
01h 10mModern paradigms of generalization, the heliocentric model of Aristarchus,...
36mBayesian Deep learning with 10% of the weights
01h 02mLarge-Scale Private Learning, Part I
48mPredicting from Noisy and Incomplete Data: Some Perspectives from Computational Learning Theory
35mUncertainty in deep learning
39mA PAC Approach to Application-Specific Algorithm Selection
42mWhat (Doesn't) Make Learning Algorithms Generalize?