Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
summary
Practical recipes: build evals that grade actual claims, not topics — combine LLM-as-a-judge with human-in-the-loop and cheap heuristics; use encoder-only models are ~10x cheaper for high-volume checks and reserve LLMs for nuanced scoring; validate graders on a golden dataset and measure surprise (Duolingo runs ~20 evals per trace); instrument agents with OTEL, evaluate trajectory and control-flow failures across traces, use log-prob for confidence, and deploy automated meta‑prompting to shorten the iteration loop.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:05Why scalable LLM evals matter
- 3:53Observability and evals
- 7:43Types of evals beyond judge
- 10:31Evaluating complex workflows
- 13:18Agent evaluation challenges
- 16:30Trajectory and pathing evals
- 18:15Inline evals vs guardrails
- 20:43Async and distributed evals
- 22:33Confidence scores
- 23:34Automating prompt optimization
related talks
35mEvaluating LLM systems: the nuts and bolts
32mA Practical Guide to LLM Evaluation
38mMaking LLMs reliable - A practical framework
28mTrustworthy LLMs: Vibe checks are not all you need
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
46mEnhancing Agent Reliability Through Evaluation-Driven Development
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
25mEvaluating An LLM's Ability To Code