The Power and Limitations of Kernel Learning
summary
Argues that kernel machines reproduce many modern ML phenomena but suffer from nearly exponential eigenvalue decay that makes RKHS problems extremely ill‑conditioned, so plain gradient descent needs astronomically many iterations to fit non‑smooth targets. Proposes a spectral flattening / preconditioner built via subsampled Nyström (called EigenPro preconditioner) that costs a small cubic on the subsample, adds ~15–30% per‑iteration overhead, and yields 5–30× speedups and comparable or better accuracy than fully‑connected nets on several benchmarks; generalization of interpolating solutions remains unresolved.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:25The limits and power of kernels
- 4:40"Shallow"/kernel architectures
- 5:42Kernel learning for modern ML
- 6:43Kernel methods for big data
- 8:41The limits of kernels
- 11:25Eigenvalue decay
- 14:50Eigenpro: practical implementation
- 16:48Comparison with state-of-the-art
- 25:23Understanding SGD
- 26:01Batch size for parallel computation
- 27:28Overfitting with kernels
- 28:57Kernel overfitting/interpolation
- 30:07Accelerated methods for kernels
- 31:14Parting Thoughts
related talks
01h 22mGeneralization IV
46mKernel and Deep Regimes in Overparameterized Learning
50mFrom Classical Statistics to Modern Machine Learning
44mConcentration Inequalites in Learning with Kernels
46mLearning Functions and Sets with Spectral Regularization
01h 04mLearning Staircases
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
01h 12mWhen is Scale Enough?