Representation Costs of Linear Neural Networks: Analysis and Design
summary
Derives closed forms for linear networks: fully connected depth D gives representation cost proportional to D·sum σ_i^{2/D} (σ_i singular values); diagonal and full-width conv nets induce Fourier-domain group norm L_{2/D}; restricted-width convs sit between L1 and L2 and shrink with filter width Q but lack a closed form; residual stacks interpolate from L2-like for small coefficients to L1-like for large ones. Uses SVD and discrete Fourier transforms; proves limitations: 2/p must be integer to realize Lp with homogeneous parametrizations.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:51Overparameterized class of models
- 3:01Interesting empirical observations
- 3:58Generalization as a double descent curve
- 5:51Empirical observations and double descent curve
- 6:33Minimum norm interpolation in overparameterized regime
- 7:19Learning as optimization (Implicit regularization)
- 8:03L2 Regularization on weights
- 11:57Representation Cost and induced complexity measure
- 13:57Analysis and Design
- 15:17Linear Fully Connected Neural Network
- 18:30Diagonal Neural Networks
- 19:52Full Width Convolutional Neural Networks
- 21:07Restricted Width Convolutional Neural Networks
- 24:06Linear non-homogenous Residual Neural Network
- 28:38Mixing depths
- 32:50Summary of Results
- 33:53Future work
related talks
12mDistilling models using the linear representation hypothesis
43mThe Linear Algebra of Deep Learning
34mThe Linear Algebra of Deep Learning
11mOn Implicit Regularization in Deep Learning
59mTutorial: Statistical Learning Theory and Neural Networks I
28mTowards the Universal Law of Robustness for Deep Neural Networks
08mNeural Networks Demystified: Part 4
49mComputation in Very Wide Neural Networks