Exact Asymptotics and Universality for Gradient Flows and Empirical Risk Minimizers
summary
Paper with Michael Celentano and Chen Chen proves that in proportional high‑dim limits one coordinate trajectory of high‑dimensional gradient flow has a deterministic limit described by Dynamical Mean Field Theory, with the joint law at finitely many times given by a Gaussian parameter process plus a nonlinear residual driven by five deterministic kernels. Assumptions: iid sub‑Gaussian X and n/d → δ and Lipschitz time‑varying regularizers. Proof discretizes flow, shows gradient descent ≈ AMP reduction and uses universality to take step size → 0.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:30Empirical Risk Measure
- 5:30Neural Network
- 7:30Gradient Flow
- 11:30Negative Perceptron
- 25:12Generalization
- 35:47Fixed Point Equation
- 40:17The Subject
- 43:47Gradient Descent
related talks
01h 00mMin-Max Optimization from a Dynamical Systems Viewpoint
12mApproximating Distributions Using Well-Conditioned Normalizing Flows
59mOptimal Transport and PDE: Gradient Flows in the Wasserstein Metric
31mGradient Descent-Ascent Provably Converges to Strict Local Minmax Equilibria with a Finite Timescale
42mWhat (Doesn't) Make Learning Algorithms Generalize?
41mTransfer learning via local convergence rates of the nonparametric least squares estimator
46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
42mA Primal-dual Analysis of Margin Maximization by Steepest Descent Methods