Information Measure Estimation and Applications: Boosting the Sample Size from n to n log n
summary
Claim: minimax estimators achieve the same error with n samples that the plug-in MLE needs with about n log n samples, an effect called effective sample size enlargement. Method: split empirical frequencies at a threshold log n/n into a “non-smooth” region and a smooth region, replace the functional near zero by a low‑order best polynomial approximation (degree ~ log n) and unbiasedly estimate its coefficients. Results: provable minimax rates for Shannon entropy, power sums f_alpha and L1 divergence, practical code and large empirical gains (multi‑fold MSE and learning improvements); multivariate best‑approximation computation remains challenging.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:37Problem Setting
- 1:37What was known?
- 2:32Classical problem
- 3:13Is there something missing?
- 3:53Our evaluation criterion: minimax decision theory
- 5:12Shall we analyze MLE non-asymptotically?
- 6:15Can we reduce the bias?
- 8:47Systematic improvement
- 9:02Start from first principle
- 10:44Minimize the maximum bias
- 13:27Approximation theory in functional estimation
- 14:31A class of functionals
- 16:04Sample complexity perspectives
- 16:36Unique features of our entropy estimator
- 18:01Adaptive estimation
- 19:56Our estimator is adaptive!
- 21:56Understanding the generality
- 23:14Applying our general methodology
- 24:37Significant difference in phase transitions
- 25:30Learning Tree Graphical Models
- 25:56Hindsight
related talks
41mTransfer learning via local convergence rates of the nonparametric least squares estimator
01h 05mDistribution Estimation and Applications III
53mOptimality in Locally Private Estimation and Learning
01h 02mSampling for Linear Algebra, Statistics, and Optimization II
48mExact Minimax Estimation for Phase Synchronization
45mOrthogonal Statistical Learning
15mFundamental Limits Of Learning In Data-Driven Problems
56mSparsifying Generalized Linear Models