The Importance of Better Models in Stochastic Optimization...
Talkat Robust and High-Dimensional Statistics (Simons Institute workshop, 2018) · 2018-10-29..11-02·
summary
Experiments compare stochastic gradient to stochastic proximal point and simple truncated models and claim proximal/truncated updates remain stable across about five orders of magnitude of initial step size while SGD only works in a narrow window; a linear regression example with condition number 15 shows SGD transiently explodes but proximal updates provably never diverge (square-summable steps) and yield asymptotically optimal covariance; on CIFAR a preliminary run reports 86% test accuracy for the alternative updates.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:35The Problem
- 6:53Models in Optimization
- 8:07Generic Optimization Model
- 9:04Stochastic Gradient Method
- 10:37Conditions
- 12:00Models
- 12:57Alternatives
- 15:09Robustness Stability
- 16:33Fog Theorem
- 23:00Weak convexity
- 23:36Local asymptotic minimax theorem
- 27:23Easy problems
- 30:59Sharp growth problems
- 36:36Experiments
- 41:19Conclusion
related talks
58mFirst-Order Stochastic Optimization
01h 01mRobustness, Stochastics, Uncertainty 2
13mNon-Parametric Convergence Rates for Plain Vanilla Stochastic Gradient Descent
01h 18mOptimization I
01h 07mStochastic Second Order Optimization Methods I
05mNon-monotonic convergence of gradient descent with large stepsize
33mStochastic Optimization for Big Data Machine Learning Problems
01h 07mOptimization Crash Course (continued)