Linear Predictors in Reinforcement Learning: ...

on YouTube ↗11m

" summary

Shows realizability can fail dramatically: there exist constructions giving an exponential lower bound on learnability even with exact linear Q representation, infinite data and good conditioning. Adding the RL-specific Bellman closure assumption yields a polynomial, statistically optimal upper bound. Presents a computationally tractable pessimistic actor-critic algorithm that solves a max–min objective using confidence sets, achieves standard O(C / sqrt(n)) sample scaling with high probability, and is minimax optimal under the stated assumptions.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 0:16What is Offline RL?
3. 1:20Overview Setting: Linear Action Value Function
4. 2:20Lower Bound
5. 5:26Goal and Intuition Idea: be robust in the face of uncertainty
6. 7:24Max-Min formulation for Offline RL
7. 8:28Main Result
8. 8:57Open Questions

" related talks

• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 48mOffline Reinforcement Learning With Only RealizabilityJason Lee · Adversarial Approaches in Machine Learning 2022
• 01h 11mFoundations of RLRayadurgam Srikant · Data-Driven Decision Processes Boot Camp 2022
• 54mOn the Hardness of Reinforcement Learning With Value-function ApproximationNan Jiang · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 44mBeyond Worst-Case: Instance-Dependent Optimality in Reinforcement LearningMartin Wainwright · Reinforcement Learning from Batch Data and Simulation
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 41mBetter Learning from the Past: Counterfactual / Batch RLEmma Brunskill · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)