Large Scale Stochastic Training of Neural Networks
summary
Claims a Hessian-driven method that Hessian backpropagation plus adaptive batch size (ABSA) and adversarial/robust regularization matches or improves accuracy while cutting SGD updates by several×. Implementation uses power iterations (~10) on the Hessian (computed only rarely, e.g. end of epoch) and low-rank/block approximations to reduce cost. Results: CIFAR‑10/SVHN/ImageNet experiments show equal or better top‑1 accuracy; ResNet on ImageNet keeps ~70% while dropping iterations from ~450k to ~66k and overall training wall‑time from ~16ks to ~3.4k s.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:38Highlevel Outline
- 2:46Stochastic Gradient Descent
- 6:36Large Batch Size
- 6:55Example
- 9:22Deep Diver
- 10:07Landscape of Loss
- 11:43Training
- 12:59Method
- 13:47Results
- 14:03Does it work
- 16:33Street View House Number
- 18:56Imagenet
- 19:26Alex
- 20:14Resonant
- 21:13Robust Training
- 24:01Max Problem
- 28:52Robust Optimization
- 29:57Conclusion
related talks
01h 04mOptimization for Machine Learning II
16mSDE Approximations and Scaling Rules for Distributed Deep Learning
33mStochastic Optimization for Big Data Machine Learning Problems
01h 24mAdversarial Examples
01h 01mRobustness, Stochastics, Uncertainty 2
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
41mOptimization Algorithms for Deep Learning
52mSplitting Gradient Descent for Incremental Learning of Neural Architectures