Is Minimax Optimization the Right Framework to Understand Learning in Minimax Games?

on YouTube ↗47m

" summary

Shows that Adam’s edge in GANs comes from its step-norm, not direction: reproducing Adam with a normalized SGD variant (AdaLR/AdaDIR) matches or beats Adam, and theory on a sparse linear generator plus cubic discriminator proves SGDA collapses to the mean while normalized SGDA recovers both modes. Recasts adversarial training as distributional robustness under an L∞ Wasserstein ball, uses entropic regularization to get p(x̃) ∝ exp(λ loss) and Langevin sampling, and reports 10×–30× speedup* on CIFAR/MNIST with the caveats of memory and augmentation handling.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 0:31Outline
3. 2:32Training Dynamics
4. 3:35Principled optimization for minimax games
5. 4:51An inconvenient truth
6. 8:34What does Adam do?
7. 10:24What we did
8. 16:29Analysis of Normalized SGDA VS SGDA
9. 20:04Illustration: SGDA mode collapse (learn the average)
10. 21:10Our Results: normalized SGDA
11. 22:43Illustration normalized SGD learns both modes
12. 25:55Adversarial examples
13. 26:37Training robust models
14. 29:58Distributional robustness perspective
15. 31:59Insight from optimal transport
16. 35:01Results: MNIST
17. 36:22Results: CIFAR
18. 37:33Conclusion (part 2)

" related talks

• 01h 00mTutorial on Deep Learning IVRuslan Salakhutdinov · Foundations of Machine Learning Boot Camp 2017
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 01h 17mImplicit Regularization INati Srebro · Deep Learning Boot Camp 2019
• 44mRobust Estimation and Generative Adversarial NetsChao Gao · Robust and High-Dimensional Statistics (Simons Institute workshop, 2018)
• 01h 01mRobustness, Stochastics, Uncertainty 2Sasha Rakhlin, Ben Recht, Laurent El Ghaoui · Bridging Continuous and Discrete Optimization Boot Camp 2017
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 11mNonparametric Density Estimation and Convergence of GANs under Besov IPMsAnanya Uppal · Meet the Fellows Welcome Event