A Primal-dual Analysis of Margin Maximization by Steepest Descent Methods
summary
Proves that steepest-descent dynamics on exponential/logistic losses drive linear predictors toward the max‑margin direction: norms diverge (roughly like log T) and each unit of norm yields one unit of norm → γ margin so the normalized classifier converges. Uses gradient flow/descent smoothness bounds and a dual formulation showing a dual mirror‑descent / KL power‑method that concentrates weights on support points. Gives empirical CDFs (including AlexNet) that margins shift right after normalization, notes extensions to deep linear/ReLU with technical caveats and open rates for partial‑data margins.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 3:22Presentation Plan
- 4:12Margins
- 7:39Margin CDF
- 12:24Concrete Bounds
- 16:55Optimization Question
- 18:11Three Essential Techniques
- 18:51Example
- 19:09Notation
- 20:10Intuition
- 23:33Proof
- 28:35Nonseparable
- 28:57Bounds
- 37:51Nonlinear Comments
related talks
01h 23mImplicit Regularization II
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
01h 00mMin-Max Optimization from a Dynamical Systems Viewpoint
01h 01mRobustness, Stochastics, Uncertainty 2
13mThe dynamical view of machine learning: implicit biases of large learning rates
46mSmooth Nonconvex Min-Max Optimization
13mNon-Parametric Convergence Rates for Plain Vanilla Stochastic Gradient Descent
01h 03mOverview of Statistical Learning Theory Part 2