Reinforcement Learning with Human Feedback

on YouTube ↗29m

" summary

Shows the actual pipeline: collect multiple model outputs per prompt and human-rank them (often 3/2/1), train a value network learns human scores as a reward model, then fine-tune the generative policy with PPO using the PPO surrogate ratio (policy probability / previous probability) so increases that already grew get amplified while decreases are damped. Notes that human labeling is the largest cost and that DPO provides a direct loss that bypasses RL by joining reward and policy training into one objective.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 1:45Large Language Models(Transformers)
3. 4:15How to fine-tune them with RLHF
4. 15:50Quick intro to reinforcement learning
5. 19:51PPO (reinforcement learning technique to fine-tune LLMs)
6. 26:04DPO(non-reinforcement learning technique to fine-tune LLMs)

" related talks

• 14mHuman-in-the-Loop Reinforcement LearningPieter Abbeel · Ray Summit 2021
• 46mEfficient Policy Optimization Techniques for LLMsKianté Brantley · The Future of Language Models and Transformers 2025
• 19mTraining Agentic ReasonersWill Brown · AI Engineer World's Fair 2025
• 53mIterative preference learning methods for large language model post trainingWei Xiong · Emerging Generalization Settings 2024
• 38mReinforcement Learning - What it is and How it WorksJoshua Starmer · Uphill Conf 2025
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 30mReinforcement Learning - ChatGPT, Playing Games & MoreDean Wampler · GOTO Chicago 2023
• 42mReinforcement Learning using Generative Models for Continuous State and Action Space SystemsRahul Jain · Reinforcement Learning from Batch Data and Simulation