Statistical Physics and Computation in High Dimension
Talkat Probability, Geometry, and Computation in High Dimensions Boot Camp (Simons Institute, Fall 2020) · 2020-08-19..08-28·
summary
Analyzes teacher-student neural ensembles (committee machines) with analytic replica solution and matching simulations to produce exact learning curves as a function of sample-to-dimension ratio α. Shows a sharp specialization transition where hidden units lock onto features above a critical α, quantifies regimes where single-layer models are solvable while deeper two-layer cases exhibit new computational gaps, and uses generative teachers plus numeric experiments to demonstrate when Bayes-optimal recovery is possible and when training algorithms get stuck.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:40ADDING HIDDEN UNITS
- 2:33GRAPHICAL MODEL
- 7:43SPECIALISATION TRANSITION
- 11:11COMPUTATIONAL GAP
- 13:12TWO-(EXTENSIVE)LAYERS PERCEPTRO
- 15:29GOING DEEP (MULTI-LAYER)
- 17:33GENERATIVE PRIORS
- 20:12SOLVABLE CASE
- 21:20KEY OBSERVATION ABOUT G-AMP
- 23:09MULTI-LAYER GENERALISED LINEAR ESTIMATION
- 24:45MULTI-LAYER AMP
- 28:34EX: PHASE RETRIEVAL y = |fx
- 31:22A "LEGO" PRINCIPLE
- 35:34DATA ON MANIFOLDS
- 36:11HIDDEN MANIFOLD MODEL
- 38:29HIDDEN MANIFOLD VS MNIST
- 40:18MNIST VS HIDDEN MANIFOLD
- 43:52SOLVING THE HMM
- 45:24GAUSSIAN EQUIVALENCE
- 48:10PHASE DIAGRAM
- 49:55RANDOM FEATURES
- 51:11Over-parametrization /d-3
- 52:54Phase transition of perfect separabili
- 55:24Gaussian v.s. orthogonal features
- 56:21DEEP GENERATIVE MODELS
- 57:20Deep convolutional GAN with random weights
- 57:56Deep convolutional GAN: ODE vs Simulation
- 58:53The realNVP: normalising flows
related talks
01h 07mTheory of Computation II
48mLearning by Local Entropy Maximization
01h 08mTutorial: Methods from Statistical Physics III
01h 11mHidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
58mStrong Generalization from Small Brains and No Training Data
12mTheoretical understanding of learning through the computational lens
36mScaling Laws, Emergent Behaviors, and AI Democratization
48mComputational-Statistical Tradeoffs in Robust High-Dimensional Estimation