Human-in-the-Loop Reinforcement Learning
summary
A pipeline converts human pairwise preferences over rollouts into a learned reward model (Bradley Terry style, trained with cross entropy) and then runs standard deep RL (SAC) to optimize that reward, with unsupervised pretraining to make early comparisons meaningful. Targets show directional cartpole behavior in under five minutes with 40 queries, an ant leg lift in ~30 minutes with 200 queries, and a backflip learned; performance approaches an oracle reward and avoids obvious reward hacking.
AI-written summary · about
chapters (11)
- 0:00Introduction to talk
- 0:42Recent progress in deep RL
- 3:08Challenges with reward specification
- 4:33Human input as reward alternative
- 5:17Pebble algorithm overview
- 6:20Training reward model from preferences
- 8:26Iterative human-in-the-loop process
- 9:10Unsupervised pre-training benefits
- 9:31Teaching novel behaviors with preferences
- 10:58Avoiding reward exploitation
- 12:48Summary and future directions
related talks
29mReinforcement Learning with Human Feedback
28mPreference Learning from Minimal Human Feedback for Interactive
35mIndustrial Applications of Reinforcement Learning
30mReinforcement Learning - ChatGPT, Playing Games & More
01h 17mTraining intelligent game agents using deep reinforcement learning
31mDesigning game AI with reinforcement learning
26mHacking Reinforcement Learning
18mMaximizing Luck in Reinforcement Learning