Hanie Sedghi, Research Scientist, Allen Institute at MLconf Seattle 2017
summary
Proposes an algorithm that recovers first-layer weight columns by forming cross-moments between labels and a third-order score function (mth-order derivatives of log-density), estimating that score with a denoising autoencoder, then doing tensor CP decomposition (tensor power iteration) to extract components. Claims polynomial sample complexity and a 1/n squared-error term plus approximation error tied to the function’s Fourier moment; reports empirical gains—10% better on MNIST and 6–12% higher accuracy when labels are downsampled versus SGD/Adam.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:18Training Neural Networks
- 0:50Convex vs. Nonconvex Optimization
- 1:39Toy Example
- 2:44Example: bad local optima
- 4:27Outline
- 4:57Three Main Components
- 6:07Background: Tensor Decomposition
- 7:41Method-of-Moments for Neural Networks
- 10:01Moments of a Neural Network
- 12:35Derivative Operator: Score Function Transformations
- 14:21NN-LIFT: Neural Network-Learning using Feature Ter
- 15:32Estimation Error Bound
- 17:11Approximation Error
- 18:17Our Main Result
- 19:09Experiment: NN-LIFT vs. Backprop
- 20:16Experiment Results: NN-LIFT vs. Backprop
- 21:44FEAST: Feature ExtrAction using Score function Tene
- 22:56Guaranteed Training of Recurrent Neural Networks
- 26:41Conclusion and Future Work
related talks
53mTensor Decompositions for Learning Latent Variable Models II
22mAnima Anandkumar, Electrical Engineering and CS Dept, UC Irvine
58mTensor Decomposition, Sparse Representations and Applications
52mLearning Mixture of Gaussians using Moment Tensor
28mConstructing Informative Features for Discriminative Learning
01h 11mTensor Decompositions: Uniqueness and Smoothed Analysis
56mTensor Decompositions for Learning Latent Variable Models I
52mTensor Methods for Learning Latent Variable Models: Theory and Practice