Holistic Evaluation of Language Models (HELM)
summary
HELM is an open-source Python framework plus public leaderboards that run reproducible model evaluations; it executes 42 benchmarks on 30 models using roughly 12 billion tokens and 17 million queries (about $38k API cost and 20k GPU hours). Coverage rises from 18% to 96% across a growing suite (19 leaderboards, 150+ datasets, 350+ models). Raw logs expose refusals, data leakage and cheating; results show MMLU saturation, prompt sensitivity that flips rankings, and gaps for long-horizon, professional, and non‑Western evaluations.
AI-written summary · about
chapters (22)
- 0:00HELM project intro
- 1:02Foundation models 2022 risks
- 2:03HELM origins and evaluation
- 3:48Filling HELM evaluation matrix
- 4:29Holistic evaluation principles
- 5:53HELM open-source toolkit
- 6:56Keeping leaderboards relevant
- 8:44HELM beyond text-only
- 11:17Specialized domain evaluations
- 13:07Safety for frontier models
- 14:10Government regulation compliance
- 15:14Missing complex behavior evals
- 16:22Professional industry evaluations
- 16:42Non-Western culture evaluations
- 17:23User harms and LM delusions
- 17:44Transparency via raw logs
- 19:51Reproducibility challenges
- 23:23Prompt variations and rankings
- 24:29Insufficient model card info
- 25:35Universal benchmarking standards
- 26:43Conclusion and future
- 27:26Q&A: Early LLM training eval
related talks
01h 06mThe Future of Language Models: A Perspective on Evaluation
31mChallenges and Considerations in Language Model Evaluation
16mBest Practices for Open Multilingual LLM Evaluation
32mHolistic Evaluation of Large Language Models: From References to Human Judgment
30mResponsible AI with fmeval - an open source library to evaluate LLMs
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
34mBuilding Trust with LLMs
42mAdvancements in Evaluating Large Language Model Applications