Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
summary
Provides convergence guarantees for Local SGD showing the outer optimizer controls a tradeoff between optimization error and stochastic noise, and can compensate for an overly small inner learning rate by using an outer learning rate >1; momentum rescales this effect (effective γ = γ/(1−μ)), while applying Nesterov acceleration in the outer loop provably speeds convergence in communication rounds. Experiments pretrain decoder-only Transformers on C4 (150M, 400M, 1B) with 50–500 local steps, finding Nesterov outer optimizer and schedule-free SGD (outer LR = 2 at 1B) outperform simple averaging.
AI-written summary · about
chapters (11)
- 0:00Outer optimizers in Local SGD
- 1:29Distributed training challenges
- 3:29Data parallel training limits
- 6:13Local SGD two-level structure
- 10:55Two-tier optimizers counterintuitive
- 11:58Theoretical setup for analysis
- 15:06Convergence theorem for Local SGD
- 19:23Outer learning rates and momentum
- 21:22Acceleration benefits
- 21:48Transformer pre-training validation
- 24:09Q&A
related talks
47mMeta-learning of Optimizers and Update Rules
16mSDE Approximations and Scaling Rules for Distributed Deep Learning
58mFirst-Order Stochastic Optimization
41mOptimization Algorithms for Deep Learning
01h 05mTutorial: Federated Optimization, Part III
01h 18mOptimization I
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization