Confounding-Robust Policy Evaluation In Infinite-Horizon Reinforcement Learning
summary
Claims a method to compute sharp bounds on infinite-horizon off‑policy values under unobserved time‑step confounding using a marginal sensitivity model, formulates optimization over stationary state‑occupancy ratios constrained by a partially identified estimating equation, and proves convergence of those bounds as confounded data grows. Checking set membership is a linear program but the support function becomes a nonconvex quadratic program; proposes alternating minimization / nonconvex projected gradient approximations and demonstrates tractable solves with Gurobi for small instances (3×3 gridworld) where bounds widen as policies move away from the behavior distribution.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:32Motivation
- 5:15Talk outline
- 7:53Infinite-horizon setting
- 12:25Problem Setup: Structural Restrictions
- 14:44Confounding-Robust Policy Evaluation
- 16:20Ambiguity set
- 17:28Towards robust policy evaluation
- 20:10Implications
- 21:12Computation
- 21:30Consistency
- 22:21Example
- 24:13Conclusion
related talks
09mTowards Credible and Effective Data-Driven Decision-Making
53mOff-policy Policy Optimization
31mOn the Complexity of Learning Good Policies With and Without Rewards
45mPolicy Gradient: Optimal Estimation, Convergence, and Generalization beyond Cumulative Rewards
44mBeyond Worst-Case: Instance-Dependent Optimality in Reinforcement Learning
11mLinear Predictors in Reinforcement Learning: ...
12mCausal Inference And Data-Driven Decision-Making
56mOn the Curses of Future and History in Off-policy Evaluation in non-Markov Environments