Generalization and Robustness in Offline Reinforcement Learning

on YouTube ↗54m

" summary

Proposes a model-based offline RL pipeline that fits a transition model by MLE then pessimism (constrain models near the MLE and plan against the worst-case model) and proves it competes with the best covered policy whenever a realizable model class and coverage measure hold, with sample complexity polynomial in function-class complexity (e.g. log |F|) rather than number of states. Extends to kernels and low-rank MDPs (bounds scale with rank d), and for imitation uses a learned model plus model uncertainty penalty to combine few expert demos with large offline logs — succeeds on a simulated humanoid where behavior cloning fails.

AI-written summary · about

" chapters (19)

1. 0:00Intro
2. 0:43Empirical RL for large-scale problems
3. 1:32Traditional Online RL paradigm
4. 2:57offline RL paradigm
5. 4:12Potential Applications of Offline RL
6. 6:01Why offline RL is challenging?
7. 8:44Finite Horizon MDPs
8. 9:32Offline Data Collection
9. 12:24Offline Data Coverage
10. 13:48Learning goal in Offline RL: Robustness
11. 15:14Learning goal in Offline RL: Generalization
12. 17:25A Model-based Approach
13. 19:02A Naive Model-based Approach
14. 25:46Formal Theoretical Guarantee for CPPO
15. 39:40Traditional Imitation Learning
16. 40:20Covariate shift in Imitation
17. 40:50Offline Imitation Learning
18. 44:06Explanation of MILO
19. 45:00Experiments of MILO

" related talks

• 34mUniform Offline Policy Evaluation (OPE) and Offline Learning in Tabular RLYu-Xiang Wang · Reinforcement Learning from Batch Data and Simulation
• 48mOffline Reinforcement Learning With Only RealizabilityJason Lee · Adversarial Approaches in Machine Learning 2022
• 35mOnline Learning with A Lot of Batch DataShie Mannor · Reinforcement Learning from Batch Data and Simulation
• 55mWhat Are the Statistical Limits of Offline Reinforcement Learning With Function Approximation?Sham Kakade · Mathematics of Online Decision Making 2020
• 48mPractical Model-based Algorithms for Reinforcement Learning and Imitation Learning, with...Tengyu Ma · Frontiers of Deep Learning 2019
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 31mOnline Reinforcement Learning With The Help Of Confounded Offline DataUri Shalit · Learning from Interventions 2022
• 11mLinear Predictors in Reinforcement Learning: ...Andrea Zanette · Meet the Fellows Welcome Event