How Randomized Analysis Can Help Us Do “Data Science:” Examples in Deep Learning and Graph Analysis
summary
Randomized analysis shows that with common random initializations, deep feedforward nets map different inputs to nearly the same representation: the angle between inputs collapses toward zero as depth increases, predicted by an iterated arc‑cosine kernel and observed for widths of a few hundred. Rotationally‑invariant weight models and asymptotic kernel limits supply the tools; batch‑norm and ResNets preserve structure, while SGD tends to keep the kernel invariant and Adam/RMSProp break it, explaining training difficulty from initialization.
AI-written summary · about
chapters (7)
- 0:00Introduction to data science
- 4:52Deep learning and depth
- 8:27Challenges of deep networks
- 13:25Statistical perspective
- 17:32Kernelized analysis
- 26:02Initialization implications
- 28:40Training dynamics
related talks
56mRandom Embeddings, Matrix-valued Kernels and Deep Learning
46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
01h 04mRandomized Embeddings and Neural Networks
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
43mThe Linear Algebra of Deep Learning
01h 02mTheory and Big Data
50mFrom Classical Statistics to Modern Machine Learning
01h 20mThe elusive generalization: classical bounds to double descent to grokking