RLHF for finer alignment with Gemma 3
summary
Gemma 3 uses two concrete fixes to reduce RLHF reward hacking: train the reward model from the same policy to keep RM judgments in-distribution and build an ensemble then collapse it into a single weight averaging checkpoint so one model emulates many without extra cost. The RL objective uses ordering-based reward signals instead of absolute scores to resist confident out-of-distribution boosts. Combined these let training run much longer with fewer hacked behaviors and yield improved LMC alignment metrics.
AI-written summary · about
chapters (9)
- 0:00RLHF for Gemma 3 intro
- 0:23What is RLHF
- 1:48RLHF two-stage process
- 2:30Reward hacking challenge
- 3:55Robust reward model design
- 5:41Ensemble reward models
- 7:06Better alignment RL algorithm
- 8:32Response ordering focus
- 9:13Conclusion
related talks
20mEverything I Learned Training Frontier Small Models
13mOptimizing RLHF Training for Large Language Models with Stage Fusion
19mTraining Agentic Reasoners
12mGemma: The responsible way to build
02h 52mLow Level Technicals of LLMs
17mPrivate Tuning of LLMs in Practice: From VaultGemma to Custom Fine-Tuning
34mOffline Reinforcement Learning and Model-Based Optimization
10mDemo: Using Gemma with the Hugging Face ecosystem