The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior
Talkat Bridging Prediction and Intervention Problems in Social Systems (Simons Institute, 2026) · 2026-01-12..01-18·
summary
Claim: offline one-shot API evaluations miss how models behave when personalized in chats. Empirical setup: 800 real users of ChatGPT and Google Gemini copy-paste 13 benchmark and recommendation questions from Prolific; simulated sock puppets prepend histories or profiles; a field evaluation shows far more diverse outputs, lower accuracy on multiple-choice, and many unseen answers never produced offline. A prototype browser extension collects real chat responses and aggregated summaries to scale study and surface personalization effects; simulations help but undercount diversity.
AI-written summary · about
chapters (9)
- 0:00Introduction to personalization
- 1:28Variability in implementations
- 3:35Evaluating model performance
- 6:30Using sock puppets for scale
- 10:38Implications for benchmarks
- 12:54Social dimensions of chatbots
- 16:26User preferences and transparency
- 21:00Future directions in research
- 23:29Q&A and concluding remarks
related talks
24mHow to Keep Your LLM Chatbots Real - A Metrics Survival Guide
01h 04mWhat are we thinking in the age of AI?
45mPanel Talk
48mDIY Personalization: How, when why to offer self-made AI
01h 05mIs ChatGPT (and Other AI) the Enemy of Diversity?
54mKeynote: Can you trust your (large language) model?
38mBuild Evals That Actually Matter
46mConversations with ChatGPT: illusion or reality?