Off-policy Policy Optimization

on YouTube ↗53m

" summary

Argues that naively optimizing importance-weighted expected reward is unstable and high-variance, and instead proposes entropy-regularized surrogate losses and reverse-KL surrogates that preserve global optima while making training tractable; combines those with a single unified model that outputs logits and reward estimates and uses doubly robust estimator style missing-data inference or simple top-layer Bayesian variance models; shows Monte Carlo policy iteration on logged trajectories (cart-pole up to ~400 steps) and warns that Bellman optimality with function approximation is the main source of instability.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 0:49The RL problem
3. 4:59Batch policy optimization
4. 10:14Optimization objectives
5. 21:31Supervised vs reinforcement learning
6. 33:44Missing data inference
7. 43:44Sequential decision making
8. 45:54Sequential RL

" related talks

• 45mPolicy Gradient: Optimal Estimation, Convergence, and Generalization beyond Cumulative RewardsMengdi Wang · Adversarial Approaches in Machine Learning 2022
• 41mConfident Off-policy Evaluation and Selection through Self-Normalized Importance WeightingIlja Kuzborskij · Reinforcement Learning from Batch Data and Simulation
• 09mTowards Credible and Effective Data-Driven Decision-MakingAngela Zhou · Meet the Fellows Welcome Event
• 28mConfounding-Robust Policy Evaluation In Infinite-Horizon Reinforcement LearningAngela Zhou · Algorithmic Aspects of Causal Inference 2022
• 44mBeyond Worst-Case: Instance-Dependent Optimality in Reinforcement LearningMartin Wainwright · Reinforcement Learning from Batch Data and Simulation
• 09mLearning Under RequirementsLuiz Chamon · Meet the Fellows Welcome Event
• 28mA Unifying View of Optimism in Episodic Reinforcement LearningCiara Pike-Burke · Mathematics of Online Decision Making 2020
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation