Splitting Gradient Descent for Incremental Learning of Neural Architectures
summary
Proposes a practical algorithm that grows networks by splitting neurons into two equal-weight copies when training reaches a parametric stationary point, claiming second-order loss reduction ≈ ε^2·λ_min. Optimization uses an infinite Wasserstein neighborhood to derive steepest descent in distribution space, produces a per-neuron splitting matrix S whose minimal eigenvector gives the split direction, and evaluates S·v efficiently via compute S·v via one extra autodiff pass so cost is near a backprop. Shows improved compact models vs pruning on CIFAR-10 and keyword spotting.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 0:32Optimization of Neural Networks
- 1:53Architecture Optimization
- 2:45Architecture Search
- 3:45Key Idea
- 6:10Parametric Case
- 9:02Functional Case
- 24:52Optimal Splitting Strategy
- 26:42Splitting Matrix
- 30:57Loss Function
- 34:22Algorithm
- 40:27Algorithm Visualization
- 41:17Eigen Calculation
- 43:12Experimental Results
- 43:39Visualization
- 45:42Comparison
related talks
46mGradient Descent: The Ultimate Optimizer
45mScale Learning and Reasoning Across Heterogeneous Gradients and Semantics
41mOptimization Algorithms for Deep Learning
01h 04mOptimization for Machine Learning II
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
01h 20mThe elusive generalization: classical bounds to double descent to grokking
31mLarge Scale Stochastic Training of Neural Networks
05mNon-monotonic convergence of gradient descent with large stepsize