Efficient Policy Optimization Techniques for LLMs
summary
Introduces three concrete algorithms: PPO++ mixes roll‑ins from a reference policy to boost exploration and shows a small GPT‑4 win‑rate gain over PPO; DR‑PO resets from demonstration data and outperforms PPO and offline baselines across model scales when the dataset responses are higher quality; REBEL reduces RL to relative‑reward regression (no value or reward model in memory), matches or exceeds PPO win rates while cutting wall‑clock time and peak memory to near iterative DPO, and scales to Llama‑3‑8B and Qwen‑2.5B benchmarks.
AI-written summary · about
chapters (5)
- 0:00Introduction to resets
- 5:55Resetting with reference policy
- 16:01Resetting with demonstration data
- 28:00Resetting with current policy
- 40:07Conclusion and Q&A
related talks
29mReinforcement Learning with Human Feedback
53mIterative preference learning methods for large language model post training
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
19mTraining Agentic Reasoners
19mRL for Autonomous Coding
32mImprove LLMs Alignment with Complete and Robust Preference Data
45mPanel Discussion
01h 38mTraining Large Language Models: Practices and Research Questions