Introduction to the tensor-programs framework, a PL approach that helps analyse ...
summary
Introduces Greg Yang’s tensor‑programs framework: a simple straight-line tensor language that encodes forward and backward neural computations parameterized by width and proves a master theorem that as hidden widths → ∞ every program’s output converges in distribution to a Gaussian process with computable mean and covariance. The framework covers nearly all architectures, yields inductive formulas for kernels, is used to extend initialization and some training results, and in practice matches infinite‑width behavior by widths on the order of thousands while requiring GP approximations (inducing points) for big datasets.
AI-written summary · about
chapters (10)
- 0:00Start
- 0:44Big Question Big success of deep learning. But why?
- 2:51Infinitely-wide Neural Networks Answer deep theoretical questions
- 11:07Setup Width-parameterised randomly-initialised NNs
- 22:1100-wide Bayesian Neural Networks Allow efficient posterior inference and prediction
- 28:54Example 1
- 33:18(Simplified) Syntax
- 38:00Semantics of Program P A family of deterministic functions
- 39:22(Simplified) Master Theorem Convergence to a multivariate normal
- 40:21Master Theorem in Action Defines a random var Z ER for every vector variable
related talks
49mComputation in Very Wide Neural Networks
28mNext-generation frameworks for Large-scale Machine Learning
10mDale Smith, Data Scientist, Nexidia
15mEnabling Tensor Language Model to Assist in Generating High-Performance Tensor...
30mTensor Logic: The Language of AI
17mUniversality of Neural Networks
45mTensorly: A Flexible Python Framework for Machine Learning
07mWTT: What the Tensor?