The Science and Practice of Open and Scalable LLM Evaluations
summary
Describes an open evaluation stack that versions configs and hashes results, runs hundreds of benchmarks reproducibly with repeats and confidence intervals, and exposes a CI-style gate for go/no‑go regression decisions. Shows Nemo‑Evaluator running 1,000+ examples at scale with sandboxing options and orchestration plans (Docker/ECS/Kubernetes), automatic logging and contamination checks. Gives concrete failure data: two problematic samples out of ~2,000 cause a 7‑day hang that wastes 500+ GPU hours, and explains limits and circuit breakers to prevent that.
AI-written summary · about
chapters (9)
- 0:00The state of LLM evaluations
- 0:22Improving evaluation standards
- 7:41Evaluator architecture
- 7:53Environment-based evaluation
- 13:21Achieving decision-grade evals
- 14:49Gatekeeping and regression
- 16:00Challenges in agentic evals
- 23:45Call to action and resources
- 24:39Q&A and concluding remarks
related talks
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
41mFrom Lab To Life: Practical AI System Evaluation
14mImproving Large Language Model Performance
29mPutting the data science back into LLM evaluation
14mTailor-Made LLM Evaluations: How to Create Custom Evaluations for Your LLM
35mEnsuring Reliable Evaluation Systems for Your ML
34mBuilding Trust with LLMs
20mFive hard earned lessons about Evals