Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains or the Whole Domain
summary
Shows a negative result that wide neural nets and RKHS contain arbitrarily spiky functions, so multiplicative extrapolation can be infinite unless the model class or data have extra structure. Gives a positive guarantee for decomposable additive models f(x)=Σ fi(xi): if each marginal has bounded density ratio and the training support graph has good spectral gap (bound involves the second eigenvalue inverse) then the P→Q error ratio is controlled (continuous Gaussian case gives a bound ~d/λmin(ΣP)). Proves worst‑case L∞ recovery needs exponential samples but if the ground truth is a Gaussian random field with spectral decay α>0 then L∞ is polynomially comparable to L2 using spherical harmonics and eigenvalue analysis; simulations (small Python experiments) show structured models extrapolate better.
AI-written summary · about
chapters (9)
- 0:00Extrapolation of nonlinear models
- 1:07Motivation from RL theory
- 2:09Zero-shot domain extrapolation
- 13:53Assumptions for nonlinear models
- 27:12Empirical validation
- 28:34Worst-case recovery motivation
- 34:23Random functions and recovery
- 40:27Learnability decay conditions
- 41:51Conclusion and challenges
related talks
56mSparsifying Generalized Linear Models
48mBenign Overfitting in Linear Prediction
30mML for the Real World: Distributionally Robust Extrapolation
53mDesigning Robust Learners
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
01h 22mGeneralization IV
01h 02mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2
33mThe Power and Limitations of Kernel Learning