Iterative preference learning methods for large language model post training
summary
Claims a provably efficient online iterative RLHF procedure that needs about D/ε^2 samples per batch and roughly D iterations (up to logs) to recover a KL-regularized optimal policy, and shows empirical gains over single‑iteration training using Llama3‑8B models. Implements reward models: a pairwise preference model and a multi‑head Armo scorer, uses iterative direct preference learning (DPO) instead of PPO for stability, and ships code and checkpoints as RLHFlow; notes masking external tool outputs and warns that tiny labeling shortcuts (1% biased samples) get amplified.
AI-written summary · about
chapters (13)
- 0:00Iterative preference learning intro
- 1:07Preference learning as decision making
- 4:17RLHF framework and challenges
- 9:27Reward over-optimization problem
- 10:29Online iterative RLHF framework
- 15:41Implementing iterative RLHF
- 23:58Ranking models from open data
- 26:10Direct preference optimization
- 33:41Experimental results and gains
- 37:25Multi-turn preference learning
- 44:34Open-sourcing models and data
- 48:56Key takeaways and future work
- 52:59Q&A
related talks
46mEfficient Policy Optimization Techniques for LLMs
29mReinforcement Learning with Human Feedback
01h 38mTraining Large Language Models: Practices and Research Questions
32mImprove LLMs Alignment with Complete and Robust Preference Data
19mRL for Autonomous Coding
42mPost-Training Small Language Models: the adventures of a practitioner
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
19mTraining Agentic Reasoners