Holistic Evaluation of Large Language Models: From References to Human Judgment
summary
Evaluation compares reference-based vs reference-less methods with concrete outcomes: BLEU/ROUGE show low Spearman/Kendall correlations (~0.1–0.2) to human judgments, embedding metrics like BERTScore give a token-level similarity example of 75.3%, and LLM-as-judge systems using frameworks such as G-Eval/Gal4 auto-generate chain-of-thought evaluation steps and produce much higher correlation with humans. Tradeoffs are explicit: cost and evaluator bias rise when using LLM judges, while reference methods demand expensive labeled data. Reported metrics use Spearman and Kendall correlations to quantify alignment.
AI-written summary · about
chapters (17)
- 0:00LLM evaluation importance
- 1:28Talk agenda
- 2:13Evaluating LLMs for safety
- 3:39LLM evaluation metrics
- 5:47Reference vs reference-less
- 7:17Limitations of statistical metrics
- 9:02Embedding-based metrics
- 11:07Reference-based workflow challenges
- 12:53LLM as a judge
- 14:17G-Eval framework
- 17:53Evaluation vs human judgments
- 21:05Robust evaluation strategies
- 21:50Final conclusions
- 22:54Speaker backgrounds
- 26:50Q&A on G-Eval
- 28:36Q&A on underrepresented languages
- 31:03Closing remarks
related talks
42mAdvancements in Evaluating Large Language Model Applications
01h 06mThe Future of Language Models: A Perspective on Evaluation
16mBest Practices for Open Multilingual LLM Evaluation
32mA Practical Guide to LLM Evaluation
34mScaling Trust
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
34mBuilding Trust with LLMs
14mTailor-Made LLM Evaluations: How to Create Custom Evaluations for Your LLM