Sample Starved Correlation Mining

on YouTube ↗28m

" summary

Claims that in the sample-starved regime (n fixed, p → ∞) a universal critical correlation rho_c governs when false positives explode, and that raising the screening threshold at the rate roughly p^{-1/n} (constant cn depending on distribution) holds the expected number of false discoveries nearly constant (Poisson-like). Simulations show p=100, n≈10 gives rho_c≈0.34; extrapolation shows p≈10^10 needs ~200 samples for rho_c≈0.6. Implements progressive screening (screen → downsample → OLS) and improves over SIS and lasso under bounded elliptically contoured densities and sparsity o(p).

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 1:02Applications
3. 2:26Process
4. 3:33Problem
5. 5:45Literature
6. 6:58Math
7. 9:35Control
8. 11:27Pure HighDimensional Asymptotics
9. 15:52Correlation Theorem
10. 18:15Critical Threshold
11. 20:49Screening
12. 23:18Progressive
13. 25:25Summary
14. 25:53Question

" related talks

• 40mTesting Correlation of Unlabeled Random GraphsJiaming Xu · Computational Phase Transitions (Simons Institute, 2020)
• 38mAll-or-Nothing Statistical and Computational Phase Transitions in Sparse Spiked Matrix EstimationCynthia Rush · Rigorous Evidence for Information-Computation Trade-offs
• 48mVariable Selection with Error Control: Another Look at Stability SelectionRichard Samworth · Workshop on Succinct Data Representations and Applications (Simons Institute Big Data 2013)
• 43mDifferential Privacy Meets Robust StatisticsSewoong Oh · Algorithmic Advances for Statistical Inference with Combinatorial Structure
• 49mFast Algorithms for Regression ProblemsDeeksha Adil · Optimization and Algorithm Design 2023
• 56mTesting Thresholds for High-dimensional Sparse Random Geometric GraphsSiqi Liu · Structural Results 2023
• 45mThe lower tail: Poisson approximation revisitedLutz Warnke · Random Instances and Phase Transitions (Simons Institute, 2016)
• 24mRobust Parameter Estimation For Random GraphsJayadev Acharya · Rigorous Evidence for Information-Computation Trade-offs