On Spurious Associations and LLM Alignment
summary
Claims an analytic characterization of the KL-regularized objective and shows the best-of-n sampling distribution is essentially Pareto-optimal for the win rate vs KL tradeoff; derives closed-form expressions using exponential tilting and CDF tricks and tests on Pythia 2B. Implements BonBon alignment (contrastive ratio objective plus MLE) to imitate best-of-n with far less verbosity than standard fine-tuning; best-of-n still wins but is computationally linear-cost, while BonBon recovers most gains and reduces the pathological long-answer failure modes.
AI-written summary · about
chapters (9)
- 0:00Workshop announcements
- 2:07LLM alignment problem setup
- 6:46Identifying off-target drift
- 11:17The bug hypothesis
- 14:20Best-of-n as a baseline
- 20:42Alignment formalism
- 26:58Theoretical optimality
- 35:34BonBon alignment method
- 41:26Experimental results
related talks
01h 08mLanguage Model Alignment: Theory & Algorithms
32mImprove LLMs Alignment with Complete and Robust Preference Data
01h 11mA Theory of Multi-objective Machine Learning
34mBeyond Scoring: Rethinking Ranking in the LLM Era
28mNear-Optimal Hypothesis Testing via Convex Optimization
40mThe causality trap: why your LLM sounds smart but thinks shallow
42mRethinking AI Alignment and Evaluation under Heterogeneous Preferences
01h 03mOverview of Statistical Learning Theory Part 2