Size-free Generalization Bounds for Convolutional Neural Networks
summary
Claims size-free generalization bounds for deep convolutional nets that are independent of input pixel count and hidden map sizes, giving a bound linear in total parameters and in the distance-to-initialization beta and logarithmic in the loss Lipschitz constant lambda. Proofs use spectral-norm analysis of the convolution operator (Op(K)), an efficient algorithm to compute full spectrum and project operator norms, and a VC/Guyot covering lemma. Experiments on a 10-layer all-conv VGG-style model on CIFAR-10 (32→16 downsampling) show distance to initialization falls as width grows while generalization gap stays flat. Op(K) spectral norm algorithm
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:18Generalization bounds for DNNS
- 2:50Generalization bounds for CNNS
- 3:52Summary of Our Results
- 4:19Recap: Convolution
- 4:41Convolution: A linear Operator
- 5:36Spectral Analysis of Convolution Sedghi et al., 2019
- 6:33Bounds for a basic setting
- 9:17Tools and Proof Outline
- 12:58Lipschitz Parametrization
- 18:17A more general setting
- 22:18Another Comparison: Fully connected case
- 26:07Generalization gap as a function of W
- 26:49Distance to initialization as a function of W
- 27:16Experiments: Role of input size
- 31:45Conclusion and Future Work
related talks
59mTutorial: Statistical Learning Theory and Neural Networks I
46mSketching for Proving Generalization of Support Vector Machines
59mSize (OOD) Generalization of Neural Models via Algorithmic Alignment
01h 21mTutorial: Implicit Bias I
01h 20mThe elusive generalization: classical bounds to double descent to grokking
28mTowards the Universal Law of Robustness for Deep Neural Networks
19mThe Spectrum of Nonlinear Random Matrices for Ultra-Wide Neural Networks
01h 40mBoot camp on generalization theory for graph learning