Representation Learning and Exploration in Reinforcement Learning

on YouTube ↗30m

" summary

Describes two concrete algorithms: HOMER uses a contrastive supervised reduction to decode a latent Block MDP into a tabular state space and explores with policies that provably cover H up to polynomial samples (scaling with latent states, actions, horizon, 1/eta and log |Φ|); on a 100-step combination-lock task where random actions need 10^100 episodes, HOMER stably discovers the correct states and finds reward. FLAMBE does model-based feature learning (MLE over Φ and Υ, oracle optimization/sampling) and gives total-variation guarantees and polynomial sample complexity in rank D, A, H and 1/ε.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 2:27Back in 2015
3. 3:33A latent state model: The block MDP
4. 5:24Main guarantee
5. 8:11Empirical Results
6. 9:36Block MDP pros and cons
7. 11:09Meanwhile
8. 13:19The low rank MDP
9. 14:04Block MDPs are low rank
10. 14:46Tangent: beyond decodability
11. 15:54Why study low rank MDPs?
12. 19:17Potpourri
13. 20:16A model-based algorithm
14. 22:14Learning one-step model
15. 25:22Simulation Lemma
16. 25:57Final steps
17. 26:18The landscape
18. 27:14Discussion

" related talks

• 48mPractical Model-based Algorithms for Reinforcement Learning and Imitation Learning, with...Tengyu Ma · Frontiers of Deep Learning 2019
• 01h 01mA Tutorial on Reinforcement Learning IEmma Brunskill · Foundations of Machine Learning Boot Camp 2017
• 42mReinforcement Learning using Generative Models for Continuous State and Action Space SystemsRahul Jain · Reinforcement Learning from Batch Data and Simulation
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 18mMaximizing Luck in Reinforcement LearningDaniel Han · PyTorch Conference 2025
• 31mStable Reinforcement Learning with Unbounded State SpaceQiaomin Xie · Reinforcement Learning from Batch Data and Simulation