Constructing Informative Features for Discriminative Learning
summary
Constructs a pipeline that estimates higher-order score functions from unlabeled X (via autoencoders or score matching), forms cross-moments E[y ⊗ score_k(X)] that equal expected derivatives of E[y|X] (Stein lemma extension), and applies CP tensor decomposition to recover latent weight vectors for mixtures of GLMs and first-layer weights of multilayer nets. Method yields scalable algorithms with provable identifiability and reports ~96% accuracy on preliminary single-layer experiments, and extends to deeper models.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:53Principles Behind Feature Learning
- 3:58Conditional Latent Variable Models: Two Cases
- 5:14Challenges in Learning LVMs
- 6:50Classical Spectral Methods: Matrix PCA and CCA
- 7:22Beyond SVD: Spectral Methods on Tensors
- 7:42Moment Matrices and Tensors
- 8:21Spectral Decomposition of Tensors
- 10:34Moment Tensors for Conditional Models
- 12:50Outline
- 13:05Score Function of Input Distribution
- 13:48Why Score Function Features?
- 16:40Operations on Score Function Features
- 18:34Automated Extraction of Discriminative Features
- 19:39Learning Mixtures of Classifiers/GLMS
- 20:37Learning Multi-layer Neural Networks
- 23:57Framework Applied to MNIST
- 25:17Conclusion: Learning Conditional Models using Tensor Methods
- 26:38Co-authors and Resources
related talks
30mHanie Sedghi, Research Scientist, Allen Institute at MLconf Seattle 2017
20mFeature Learning: From Optimization to Modern Generalization
01h 01mMapping multisets to vectors
37mLinear representations of concepts in modern AI models
40mYou can AI like an Expert
46mSolvable Model of Unsupervised Feature Learning
51mEfficient Retrieval of Influential LLM Training Examples
39mMachine Learning in Financial Credit Risk Assessment