Online Learning with A Lot of Batch Data
summary
Claims that large offline batches shrink online complexity: with only L of d context features observed in logs, least-squares + an optimism-based algorithm (AFLO) yields regret scaling like (d−L)√T instead of d√T. Uses tools: linear contextual bandits, importance sampling, least squares, causal diagrams and an invertibility assumption from causal inference + RL. Shows a two-state maintenance example where importance-sampling variance blows up with horizon, and defines decoupled POMDPs where off‑policy evaluation becomes identifiable and computable.
AI-written summary · about
chapters (14)
- 0:00Start
- 6:03Motivation
- 9:01Part I: Linear Bandits + Confouned Data
- 10:33Setup: Linear Bandits + Confouned Data
- 11:55Partially Observable Data = Linear Constraints
- 15:14Part II: Off-Policy Evaluation in Reinforcement Learning
- 16:07Off-Policy Evaluation (OPE)
- 17:46Example: Probabilistic Maintenance (Numeric)
- 21:32Partially Observable Markov Decision Process (POMDP)
- 26:19Off-Policy Evaluation in POMDPS
- 29:14The Three Layer Causal Hierarchy
- 30:43POMDP Limitation
- 31:24A Special POMDP (DE-POMDP)
- 33:41Conclusions
related talks
31mOnline Reinforcement Learning With The Help Of Confounded Offline Data
54mGeneralization and Robustness in Offline Reinforcement Learning
41mBetter Learning from the Past: Counterfactual / Batch RL
43mOnline Learning via Offline Greedy Algorithms: Applications in Market Design and Optimization
43mContextual Inverse Optimization: Offline and Online Learning
55mWhat Are the Statistical Limits of Offline Reinforcement Learning With Function Approximation?
44mParsimonious Learning-Augmented Algorithms
42mBypassing the Impossibility of Online Learning Thresholds: Unbounded Losses and Transductive Priors