Weak-to-Strong Generalization
summary
Paper demonstrates that fine-tuning larger GPT-4–family models on labels produced by smaller models often yields better performance than those weak supervisors — a phenomenon called weak-to-strong generalization — but never fully recovers ground-truth performance; across NLP, chess, and reward modeling PGRs range from near 0 up to ~80% on NLP with tricks. Experiments use pretrained GPT-2→GPT-4 families, bootstrapping chains, and a simple confidence loss regularizer; results help but reward modeling remains hardest.
AI-written summary · about
chapters (10)
- 0:00Introduction to the topic
- 1:00The core alignment challenge
- 4:26Experimental framework
- 8:21Results and metrics
- 12:24Baseline task evaluations
- 14:02Methodological improvements
- 20:18Understanding model behavior
- 25:37Salience and generalization
- 29:37Limitations and open questions
- 34:00Q&A and conclusion
related talks
42mTransfer learning for weak-to-strong generalization
33mBuild Better AI with First-Party Data and Fine-Tuning
01h 05mCan you trust your (large language) model?
43mState of GPT
39mCollect Data to Fine-Tune Your Large Language Model
21mBeat GPT 4 and Build the Best AI For You: Mobilizing First-Party Data with Fine-Tuning
01h 38mTraining Large Language Models: Practices and Research Questions
21mBeat GPT 4 and Build the Best AI For You: Mobilizing First-Party Data with Fine-Tuning