Rethinking AI Alignment and Evaluation under Heterogeneous Preferences
summary
Shows that fitting one reward to heterogeneous users is information-theoretically doomed: mixtures of Bradley–Terry utilities are non-identifiable and pairwise data can force any ordinal-only method to suffer nontrivial loss. Standard RLHF/DPO can incur exponential distortion under a KL constraint; leaderboards (Elo/MLE) already hide minorities. A robust alternative, Nash Learning from Human Feedback, attains the optimal distortion bound. Semi-synthetic tests on Chatbot Arena data report better local stability of pluralistic rankings versus Bradley–Terry baselines.
AI-written summary · about
chapters (12)
- 0:00AI alignment and evaluation
- 1:04Core challenges in alignment
- 4:47Reward learning from preferences
- 9:27Limitations of universal rewards
- 13:24Alignment with heterogeneous utilities
- 17:03RLHF distortion with ordinal data
- 20:54Information-theoretic limits
- 26:36RLHF distortion spectrum
- 30:57Nash Learning for robust alignment
- 32:45Fair ranking for diverse users
- 37:47Chatbot Arena experiments
- 39:36Summary and open directions
related talks
28mPreference Learning from Minimal Human Feedback for Interactive
31mDecision theoretic foundations for human-AI collaboration
47mAligning ML objectives with human values
34mBeyond Scoring: Rethinking Ranking in the LLM Era
46mEliciting and Enforcing Subjective Individual Fairness
29mReinforcement Learning with Human Feedback
25mEmbracing Real-World Pluralism: Moving beyond "gold standard" truth in AI evaluation
32mImprove LLMs Alignment with Complete and Robust Preference Data