Reinforcement Learning in Recommender Systems: Some Challenges
summary
Highlights practical obstacles and partial solutions: recommender action sets are often stochastic action sets that break standard MDPs; user interaction is a low signal-to-noise POMDP with months‑long learning (a four‑month YouTube experiment shows slow cumulative effects); proposes temporal aggregation and commitment to amplify tiny advantages; introduces a Slate Q decomposition using conditional‑logit choice models, LP relaxation (Charnes‑Cooper) and top‑k greedy serving; shows a live re‑ranking A/B yielding about 1% YouTube uplift.
AI-written summary · about
chapters (18)
- 0:00Intro
- 4:09RL in User-Facing/Interactive Systems nature RL has found tremendous success with deep models
- 5:24Some Challenges in User-facing RL (RecSys) Scale • Number of users (multi-user/MDPs) & actions combinatoriales, slates Idiosyncratic nature of actions
- 11:30I. Stochastic Action Sets
- 13:42SAS-MDPs: Constructing an MDP
- 16:07SAS-MDPs: Solving Extended MDP
- 17:44II. User-learning over Long Horizons Evidence of (very) slow user leaming and adaptation
- 24:16Advantage Amplification Temporal aggregation leg, fixed actions can help amplify advantages
- 24:52Advantage Amplification Temporal aggregation (eg, fixed actions) can help amplify advantages
- 28:19Advantage Amplification Key points
- 29:02An MDP/RL Formulation Objective: max cumulative user engagement' over session
- 31:08The Problem: Item Interaction The presence of some items on the slate impacts user response hence value of others
- 33:53User Choice: Assumptions Two key, but reasonable, assumptions
- 37:08Full Q-Learning Decomposition still holds, standard Q-leaming update
- 37:52Slate Optimization: Tractable Standard formulation: Fractional moved-integer program
- 38:37Slate Optimization: Tractable Standard formulation: Fractional mixed-integer program
- 40:43Synthetic Experiments Synthetic environment
- 46:30Robustness to User Choice Models Change user choice model to cascade Joachims 2002
related talks
41mRecommender Systems: Crafting Effective Personalised Experiences
32mLet's talk about Recommendation Systems!
43mRecommendation Systems in Market Research
23mML-Powered Search & Recommendation 101
58mRecommender Systems – A Walkthrough
37mA primer on recommendation systems.
24mImproving YouTube Recommender systems for big screen users
29mFurther enhancements of recommender systems using deep learning