Towards Understanding Generalization Properties of Score-Based Losses
summary
Connects the statistical loss of score-based fitting to the sampling cost of the inference chain: parameter error of score matching is bounded by MLE error times a factor controlled by the mixing time of Langevin and a smoothness term, with the mixer captured by a restricted Poincaré constant. Shows score matching is computationally attractive (a closed-form quadratic estimator for low-degree exponential families) while MLE gradients are provably hard, and annealed/masked losses map to faster Gibbs/tempering samplers.
AI-written summary · about
chapters (14)
- 0:00Introduction and background
- 1:53Generative modeling overview
- 5:57Energy-based modeling paradigm
- 10:42Score matching basics
- 14:17Langevin dynamics for inference
- 18:33Statistical cost of score matching
- 21:33Basic framework and challenges
- 25:33Exponential families case study
- 30:26Theoretical efficiency bounds
- 37:12Proof sketches and lower bounds
- 45:41Computational hardness and benefits
- 50:55Generalizing score matching
- 54:03Tempering and lifted diffusions
- 1:02:25Discrete models and outlook
related talks
35mLearning General Gaussian Mixtures With Efficient Score Matching
01h 02mUnexpected Test Losses from Generalization Theory?
01h 30mHigh-Dimensional Statistics II
42mWhat (Doesn't) Make Learning Algorithms Generalize?
01h 23mGeneralization II
33mTradeoffs in Large Scale Learning: Statistical Accuracy vs. Numerical Precision
41mTransfer learning via local convergence rates of the nonparametric least squares estimator
15mFundamental Limits Of Learning In Data-Driven Problems