Understanding Contrastive Learning and Self-training

on YouTube ↗45m

" summary

Analyzes InfoNCE on linear nets trained on a Gaussian mixture with an explicit augmentation model (pairs drawn from same component with probability delta). In the population limit and shared-covariance setting, optimizing InfoNCE yields a projection whose column space lies inside the Fisher subspace (if r ≥ K) and equals it when delta = 1; a CLIP-style multimodal loss yields the corresponding Fisher subspace in each modality. Separate result proves a simple self-training filter-and-retrain (remove examples where prediction disagrees with label, then retrain) strictly improves accuracy for linear classifiers on linearly separable data with label noise, with implications for recovering accuracy under label-privacy corruption.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 2:32Contrastive learning overview
3. 7:29Mathematical frameworks
4. 12:35Motivation and methodology
5. 15:01GMM background
6. 18:28SVD vs. fisher subspace
7. 20:37Fisher subspace analysis
8. 25:42Defining augmentations
9. 30:03Results for single modalities
10. 36:46Multimodal GMM results
11. 42:59Conclusion and future work

" related talks

• 17mSelf-Training Converts Weak Learners to Strong Learners in Mixture ModelsSpencer Frei · Deep Learning Theory Symposium 2021
• 01h 02mTutorial: Statistical Learning Theory and Neural Networks IISpencer Frei · Deep Learning Theory Workshop and Summer School 2022
• 01h 31mUnsupervised LearningSantosh Vempala · The Brain and Computation Boot Camp
• 01h 18mUnderstanding Generalization from Pre-training Loss to Downstream TasksTengyu Ma · Modern Paradigms in Generalization Boot Camp 2024
• 51mEfficient Retrieval of Influential LLM Training ExamplesRoger Grosse · Agency in Collaborative Learning 2026
• 01h 30mToward Understanding In-context LearningTengyu Ma · Special Year on Large Language Models and Transformers, Part 1 Boot Camp
• 01h 37mTutorial: Implicit Bias IINati Srebro · Deep Learning Theory Workshop and Summer School 2022
• 59mStudying Failure Modes of Deep Learning and Ways To Fix ThemSoheil Feizi · Adversarial Approaches in Machine Learning 2022