A Modern Maximum-Likelihood Theory for High-Dimensional Logistic Regression

on YouTube ↗44m

" summary

Claims that classical logistic MLE breaks down when p/n = κ is nonnegligible: coordinates concentrate around α·β_j (systematically inflated), their variance blows up to σ^2 (not Fisher info), and the LRT converges to c·χ^2 with c>1. Tools: an explicit three–equation system for (α,σ,λ), AMP with state evolution, leave‑one‑out and random convex geometry to locate the MLE phase transition (Cover point κ=0.5) and to predict finite‑sample corrections. Simulations and an estimator for γ recover uniform p‑values; Bartlett correction fails.

AI-written summary · about

" chapters (36)

1. 0:00Intro
2. 0:37Logistic regression in R: n = 200, p = 60
3. 1:29Logistic Regression setting
4. 2:18Basic staples of classical theory
5. 3:52Scaling
6. 5:00Unbiasedness of MLE? First Example
7. 5:58Unbiasedness of MLE? Second example
8. 6:34Consequence for predicted probabilities
9. 6:52Accuracy of standard errors?
10. 7:47Accuracy of Wilks' theorem?
11. 8:30Merely a finite sample effect?
12. 10:26Bartlett corrected p-values
13. 11:08Failures of Classical Theory
14. 11:42Asymptotic Setting
15. 13:05When does the MLE exist?
16. 13:32Analytical characterization
17. 16:08A nonlinear system of equations
18. 18:12Average' MLE behavior
19. 21:32Variance
20. 22:58Distribution of nulls
21. 23:48Distribution of the LRT
22. 24:27Bulk and tail asymptotics using our correction
23. 24:51Non-Gaussian covariates
24. 25:59Tools and Inspiration
25. 26:57Approximate message passing (AMP) for the MLE
26. 28:47The algorithm
27. 29:57Why this algorithm?
28. 31:35State evolution
29. 33:07Correctness of SE
30. 34:26Convergence to the MLE
31. 35:17Analysis of LRT
32. 36:00Leave-one-out representation
33. 36:52Main steps: LRT
34. 37:45Recap
35. 38:22Empirical performance: null LLR p-values
36. 38:37Summary and future research

" related talks

• 01h 15mAn overview of classical robust statistics and generalizations to the futurePo-Ling Loh · Modern Paradigms in Generalization Boot Camp 2024
• 57mA New Perspective on High-Dimensional Causal InferencePragya Sur · Deep Learning Theory Workshop and Summer School 2022
• 41mTransfer learning via local convergence rates of the nonparametric least squares estimatorJohannes Schmidt-Hieber · Domain Adaptation and Related Areas 2024
• 29mOn Learning Distributions from their SamplesSudeep Kamath · Information Theory, Learning and Big Data (Simons Institute, 2015)
• 53mDesigning Robust LearnersJacob Steinhardt · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 01h 04mStatistical Physics and Computation in High DimensionFlorent Krzakala, Lenka Zdeborova · Probability, Geometry, and Computation in High Dimensions Boot Camp (Simons Institute, Fall 2020)
• 29mEfficient Minimax Optimal Strategies for Universal PredictionPeter Bartlett · Information Theory, Learning and Big Data (Simons Institute, 2015)
• 31mLogistic Regression Behind the ScenesChristopher White · PyData DC 2016