Follow the Leader with Dropout Perturbations
summary
Claims a concrete result for the expert/hedge setting: in the single-neuron linear-loss case applying multiplicative dropout (zeroing components with p=1/2) each trial gives the standard optimal worst-case regret scaling (the sqrt(L log N) regime plus log N behavior) while needing no learning-rate tuning*. Proof shows constant expected loss per adversarial “sweep” and matches IID behavior when there is a gap; extensions to matrix/PCA cases are open and the skipping-whole-row variant fails in worst case.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:15Outline
- 1:10Dropout training
- 2:32Online learning with expert
- 4:08On-line learning
- 6:27Predicting with a random expert
- 7:18How do we measure performance
- 8:31Main algorithms
- 14:23How good?
- 25:39What regularization?
- 26:43Simple algorithms
- 28:48Our analysis of dropout
- 30:49Worst-case pattern
- 37:15Overview of proof for noisy case
- 37:44Outlook
related talks
42mBypassing the Impossibility of Online Learning Thresholds: Unbounded Losses and Transductive Priors
01h 01mRobustness, Stochastics, Uncertainty 2
53mDesigning Robust Learners
01h 20mThe elusive generalization: classical bounds to double descent to grokking
01h 17mGeneralization I
01h 17mOnline Reinforcement Learning and Regret
38mBest of Both World Algorithms from I.I.D. to Adversarial Data
44mBeyond Worst-Case: Instance-Dependent Optimality in Reinforcement Learning