Who Watches the Watchmen? Understanding LLM Benchmark Quality
summary
Shows that popular benchmarks like MMLU with 115,000 questions miss almost all real input space once you calculate combinatorics from a 100,000‑word vocabulary and a context window of 4,000 tokens, yielding on the order of 10^3700 possible contexts and effectively zero coverage. Uses embeddings and t‑SNE to show insurance queries land in sparsely tested regions, compares benchmark and Common Crawl coverage, and demos practical mitigations: custom task benchmarks, embedding analysis, Aria and PromptFu for feedback, graph RAG to reduce hallucination, and a public notebook.
AI-written summary · about
chapters (11)
- 0:00LLM benchmark quality concerns
- 2:11Benchmark quality matters
- 4:41Input space of LLMs
- 8:54Structured inputs coverage
- 13:13Coverage gaps in benchmarks
- 13:53Robustness in language models
- 17:25MMLU subject coverage
- 19:10Benchmark coverage visualization
- 21:38Customized benchmarks
- 24:53Improving LLM systems
- 27:42Call to action
related talks
13mOptimizing and Benchmarking Open-Source LLMs
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
28mHow to Measure Quality os LLMs
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
38mMaking LLMs reliable - A practical framework
49mBen Hall & Barbara Fusinska
40mJudgement Day: Benchmarking "Black Box" LLMs With Open Legal Datasets
10mWhat goes into the making of LLMs