Multi-Agent Reinforcement Learning In Stochastic Games: From Alphago To Robust Control

on YouTube ↗15m

" summary

Shows that a simple model-based multi-agent RL plug-in estimator achieves nearly minimax sample complexity scaling with state count S, action counts a1+a2, discount factor and error ε, and matches a lower bound up to constants. Constructs a fully decentralized Q-learning algorithm with two-time-scale updates that provably converges to Nash in two-player zero-sum stochastic games and is rational against stationary opponents. Analyzes linear quadratic games where policy optimization with a two-loop scheme attains global convergence, preserves robustness via implicit regularization, and can beat LMI/SDP solvers in practice.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 0:23Background
3. 0:39Prelude
4. 2:42Motivation
5. 3:25Our Answer
6. 5:11Recall: Why is Multi-agent RL Challenging?
7. 6:02Why is Decentralized Learning Important?
8. 7:20Decentralized Q-Learning Dynamics
9. 8:04Features of the Learning Dynamics
10. 9:24Another Fundamental Class of Stochastic Games
11. 10:19A Unified Model for Robust/Risk-sensitive Control
12. 10:51Landscape & Algorithm
13. 11:15Landscape Cont'd
14. 11:45Implicit Regularization
15. 13:08Concluding Remarks

" related talks

• 31mStable Reinforcement Learning with Unbounded State SpaceQiaomin Xie · Reinforcement Learning from Batch Data and Simulation
• 31mLearning to Control Unknown SystemsRahul Jain · Societal Networks 2018
• 01h 05mFoundations of RLRayadurgam Srikant · Data-Driven Decision Processes Boot Camp 2022
• 42mA Regret Minimization Approach to Mutli-Agent Control and RLElad Hazan · Multi-Agent Reinforcement Learning and Bandit Learning
• 37mSelfish Robustness and Equilibria in Multi-Player BanditsVianney Perchet · Mathematics of Online Decision Making 2020
• 30mLearning Multi-Agent Collaborations With DecompositionYuandong Tian · Reinforcement Learning from Batch Data and Simulation
• 42mReinforcement Learning using Generative Models for Continuous State and Action Space SystemsRahul Jain · Reinforcement Learning from Batch Data and Simulation
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond