Scaling Trust
summary
Claim: replace most large-scale human labeling with an LLM-based evaluator trained to match annotator judgments. The process builds a golden data set with strict annotation guidelines, trains a judge LLM until human agreement, then runs that judge in production to monitor metrics daily. Reported outcomes: near-human agreement, massive cost savings (human annotation ≈ $1/sentence vs tiny-model judges up to ~3,000× cheaper), concrete checks like pass-k for consistency, tool-call validity, task-completion scoring, and a dashboard + distillation to cheaper models for latency/cost control.
AI-written summary · about
chapters (12)
- 0:00Scaling trust in LLMs
- 1:12Evaluating large language models
- 4:00LLMs as judges
- 7:30High-quality judge LLMs
- 13:55Evaluating AI agents
- 16:22Blackbox evaluation
- 20:35Agent consistency
- 22:22Agent deployment triad
- 23:25Whitebox evaluation
- 26:59Agent evaluation metrics
- 28:23Evaluation in product lifecycle
- 30:54Key takeaways
related talks
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
32mHolistic Evaluation of Large Language Models: From References to Human Judgment
14mImproving Large Language Model Performance
59mWorkshop: Choosing the Best Open Source LLM for Your Application
34mBuilding Trust with LLMs
28mTrustworthy LLMs: Vibe checks are not all you need
46mEnhancing Agent Reliability Through Evaluation-Driven Development
Stop Paying for Frontier Models