Understanding the Robustness of Deep Learning
summary
Pre-trained models help robustness, but the adaptation rule matters: with limited labeled adaptation data and overparameterized features, updating all weights (fine-tuning) often distorts features used by unseen domains and hurts OOD accuracy, while linear probing (fix features, train head) preserves OOD generalization. Theory uses a two-layer linear model and gradient-flow invariants to show random head init causes that distortion; a simple recipe LPFT (probe then fine-tune) reduces feature drift and recovers the best of both, giving up to ~10% OOD gains in experiments.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:35The Challenge of Robustness
- 1:25Nurture vs. Nature in ML
- 2:47Talk Outline and Approach
- 4:43Pre-training Overview
- 5:58Pre-training for Robustness
- 8:15Types of Pre-training
- 10:42Leveraging Pre-trained Models
- 13:15Empirical Results: Breeds
- 16:43Results: CIFAR-10.1
- 19:08Fine-tuning Nuances
- 21:08Summary of Empirical Findings
- 24:08Theoretical Insights
- 29:00Theory Results
- 33:46Simplest Setting Analysis
- 37:22Feature Distortion Theory
- 41:27Best of Both Worlds: LPFT
- 45:51Evaluating LPFT
- 52:36Open Questions
- 53:53Accuracy on the Line
- 55:55Simplicity Bias
- 58:29Conclusion and Q&A
related talks
01h 27mChallenges in Making LLMs Safe and Robust
09mLearning Under Requirements
42mExploring Pillars of Trustworthy AI: Robustness and Fairness
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
01h 20mThe elusive generalization: classical bounds to double descent to grokking
36mLanguage Model Post-Training in 2025: an Overview of Customization Options Today
46mLearning Theory of Transformers: Generalization and Optimization of In-Context Learning
28mTowards the Universal Law of Robustness for Deep Neural Networks