The Future of Language Models: A Perspective on Evaluation
summary
Argues that evaluation lags model capability and gives concrete methods and results: builds a human-labeled social-media dataset of 4,000 annotated tweets and scales to 2.5M noisy tweets with a human-in-the-loop pipeline; processes 600 suicide narratives from a 270K corpus and estimates about 10% legal interaction cases using local Llama zero-shot assistance; proposes an instance-level separability-weighted ELO for long-form comparisons and a graph-based consensus merge baseline for sampled generations; explores comparing model signatures (logit/latent internals) as an alternative evaluation axis.
AI-written summary · about
chapters (12)
- 0:00Language model evaluation intro
- 2:10Why evaluation is critical
- 8:19Rethinking benchmarks and tasks
- 15:27Real-world evaluation with social data
- 23:30Evaluating sensitive tasks
- 30:37Efficiency versus depth in evaluation
- 37:14Evaluating long-form generation
- 45:52Introducing separability in evaluation
- 53:07Model variability as baseline
- 56:24Challenge of test data distributions
- 57:26Evaluating without language
- 1:01:19Q&A
related talks
32mHolistic Evaluation of Large Language Models: From References to Human Judgment
42mAdvancements in Evaluating Large Language Model Applications
51mTowards a Pragmatic Understanding of Large Language Models
40mFrom Simulated Subjectivity to Collective Consciousness in Large Language Models
40mThe Future of Language Models
31mChallenges and Considerations in Language Model Evaluation
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
29mHolistic Evaluation of Language Models (HELM)