Best Practices for Open Multilingual LLM Evaluation
summary
Advice: pick benchmarks whose difficulty still separates current models — CIV 200 (200 languages) and MultiBlim (100 languages) are too easy, MultiBlim already shows ~95–99% for LLaMA variants so it’s saturated. Avoid machine‑translated suites like EU21 (21 languages) because translation quality confounds model ability. Automatic metrics (ROUGE/BLEU or GPT‑4 judges) show poor correlation with human preference across English, Chinese and Indonesian. Use confidence intervals or standard error (10 runs changes model ranking) and LM Evaluation Harness for replicable, per‑language reporting. A new multilingual open‑ended QA benchmark is under development.
AI-written summary · about
chapters (11)
- 0:00Multilingual LLM evaluation intro
- 1:04Selecting multilingual benchmarks
- 4:22Avoiding machine translations
- 6:10Fair open-ended task eval
- 8:17Cultural adaptation in benchmarks
- 9:42Tasks aligned with human judgment
- 10:45Confidence intervals for models
- 11:49Consistent and replicable eval
- 13:15Disaggregating benchmark results
- 14:19Creating new multilingual benchmarks
- 15:23Summary and resources
related talks
57mMachine Translation of Human Languages in the Age of LLMs: Is This the End of the Language Barrier?
32mHolistic Evaluation of Large Language Models: From References to Human Judgment
31mChallenges and Considerations in Language Model Evaluation
32mA Practical Guide to LLM Evaluation
42mAdvancements in Evaluating Large Language Model Applications
33mPractical Evaluation of LLMs for Code
44mKeynote: The Art of Language: Mastering Multilingual Challenges in LLMs
44mKeynote: The Art of Language: Mastering Multilingual Challenges in LLMs