The Mean-Squared Error of Double Q-Learning
summary
Claim: averaging the two maintained Q estimates and using twice the step size (2G/(n+1)) makes Double Q-learning match the asymptotic mean-squared error of standard Q-learning. Analysis uses linear function approximation and linear stochastic approximation plus Lyapunov equations to compare covariances; updates flip a Bernoulli coin to choose which Q to update. Empirical checks on Baird’s example, Gridworld and neural nets show improved transients and that using only one Q estimate is strictly worse.
AI-written summary · about
chapters (7)
- 0:00Start
- 4:55Maximization Bias (van Hasselt, 2010)
- 11:45Goals
- 14:29Double TD Learning
- 15:17Asymptotic Mean-Squared Error
- 22:41Conclusions
- 31:42How to Fix the Problem?
related talks
11mData Efficient Methods for Reinforcement Learning
01h 30mHigh-Dimensional Statistics II
42m"Federated Reinforcement Learning: Statistical and Communication Trade-offs
45mPolicy Gradient: Optimal Estimation, Convergence, and Generalization beyond Cumulative Rewards
16mSDE Approximations and Scaling Rules for Distributed Deep Learning
33mTradeoffs in Large Scale Learning: Statistical Accuracy vs. Numerical Precision
01h 02mEstimation Beyond the IID Setting (Two Vignettes)
15mAccelerQ: Accelerating Quantum Eigensolvers With Machine Learning on Quantum Simulators