Evaluating LLM systems: the nuts and bolts
summary
Argues that systematic evaluation is the key to shipping reliable LLM products, showing concrete methods: use LLM-as-judge (e.g. GPT‑4 scoring GPT‑3.5), hundreds of labeled test cases and heuristics, Chain‑of‑Thought summaries, structured outputs ingested to a DB, and CI/CD hooks for offline tests plus an online dashboard for real‑time monitoring. Demonstrates catching real failures (car dealership price exploit, McDonald’s bizarre recipes), measures spikes in thumbs‑up/down and conversation length, and iterates until metrics actually improve.
AI-written summary · about
chapters (17)
- 0:00LLM evaluation challenges
- 2:54Poor LLM evaluations impact
- 4:18Traditional testing fails for LLMs
- 6:22Why robust LLM evaluations matter
- 7:46Three LLM evaluation strategies
- 9:11Human judgment in LLM eval
- 11:59Using another LLM to evaluate outputs
- 13:22Offline vs online evaluations
- 16:31Designing offline evaluations
- 19:04Building evaluation datasets
- 21:32Online evaluations after deployment
- 24:02Scaling online evaluations
- 26:09LLM-as-judge for feedback
- 28:59Aligning with product needs
- 31:26Closing loop between evals
- 32:54Integrating LLM evaluations
- 33:59Evaluations as competitive edge
related talks
38mMaking LLMs reliable - A practical framework
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
32mA Practical Guide to LLM Evaluation
14mTailor-Made LLM Evaluations: How to Create Custom Evaluations for Your LLM
29mCustomizing and Evaluating LLMs, an Ops Perspective
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
25mEvaluating An LLM's Ability To Code
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications