Online Reinforcement Learning With The Help Of Confounded Offline Data

on YouTube ↗31m

" summary

Two linked papers claim offline, confounded logs can speed online learning under concrete assumptions. For linear contextual bandits the method estimates linear constraints from logged context–action–reward triplets and, if allowed to query the logging policy, provably reduces regret from O(d sqrt(T)) to O((d−l) sqrt(T)) by effectively removing observed-context dimensions; the bound degrades if ROH is hard to estimate. For RL the authors derive an unbiased, GAN‑like objective and two algorithms (a slow provable one and a practical gradient version) and test on Assistive Gym, showing faster learning under moderate context shift but failure under large shifts and no proven general acceleration yet.

AI-written summary · about

" chapters (5)

1. 0:00Motivation and context
2. 3:24Linear bandit research
3. 13:33Imitation learning analysis
4. 23:57Reinforcement learning aid
5. 28:24Conclusion and discussion

" related talks

• 35mOnline Learning with A Lot of Batch DataShie Mannor · Reinforcement Learning from Batch Data and Simulation
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 43mContextual Inverse Optimization: Offline and Online LearningIlan Lobel · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 34mUniform Offline Policy Evaluation (OPE) and Offline Learning in Tabular RLYu-Xiang Wang · Reinforcement Learning from Batch Data and Simulation
• 43mOnline Learning via Offline Greedy Algorithms: Applications in Market Design and OptimizationNegin Golrezaei · Data-Driven Decision Processes Boot Camp 2022
• 41mBetter Learning from the Past: Counterfactual / Batch RLEmma Brunskill · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 48mOffline Reinforcement Learning With Only RealizabilityJason Lee · Adversarial Approaches in Machine Learning 2022