Evaluation is all you need
summary
Maps the open‑source evaluation landscape, compares techniques like RAGAS and Evaluation Driven Development and frameworks such as DeepEval and Phoenix and LangFuse and braintrust, and shows concrete combos of tests, RAG‑specific evals, and observability. Presents real failure examples (Chevy $1 pricing, glue on pizza, fabricated legal citations), a production email automation that uses screenshots to gain trust and works in practice, runtime LLM‑as‑judge checks at up to ~3,000 tokens/sec, and a decision checklist, code patterns, and a playbook tied to business metrics (2.3 average metric cited).
AI-written summary · about
chapters (6)
- 0:00LLM evaluation challenges
- 5:50Why evaluation is critical
- 11:57Classic vs modern methods
- 17:44Designing evaluation processes
- 25:54Eval-driven AI systems
- 31:41Key takeaways & Q&A
related talks
49mEvals 101
20mFive hard earned lessons about Evals
46mEnhancing Agent Reliability Through Evaluation-Driven Development
35mEvaluating LLM systems: the nuts and bolts
38mMaking LLMs reliable - A practical framework
31mEvaluating RAG Applications: From Retrieval to Response Quality
42mAdvancements in Evaluating Large Language Model Applications
31mBeyond Vibes: Building Evals for Generative AI