Testing What Matters: Evaluating LLM Relevancy with DeepEval
summary
DeepEval is a Python library that runs LLM test cases through a judge model using the G-Eval technique, installs via UV/virtualenv and calls model APIs (demo uses Google Generative AI Flash 3.6). Tests score 0–1 with configurable thresholds (demo uses 0.8 and a repo example at 0.7), some runs return score 1.0 while format-mismatched outputs show None scores, and DeepEval reports plug into CI/CD; demo exercises an answer relevancy metric and safety/agent metrics.
AI-written summary · about
chapters (22)
- 0:00Setting the Stage: 'Sounds Right' and 'Is Right' Are Two Different Things
- 1:05Speaker Intro - Monika Sharma, Software Engineer, Salesforce
- 1:35Agenda: Why LLM Testing Is Needed and How DeepEval Works
- 2:07Why Deterministic String and JSON Assertions Break on LLM Output
- 3:38What DeepEval Is - 'Pytest for LLMs' and the G-Eval Technique
- 4:40How DeepEval Works: Test Cases, Metrics Engine, Judge Model, Reports
- 5:43The Metrics Taxonomy: RAG, Agent, Safety, Custom and Conversational
- 6:14RAG Metrics: Faithfulness, Answer Relevancy, Contextual Precision and Recall
- 6:47Agent Metrics: Task Completion and Tool Correctness
- 7:18Safety and Compliance: Toxicity, Bias, PII Leakage, Role Violation
- 8:48Installation and Setup: Python, UV, API Keys, Project Structure
- 13:23Writing the First Test Against an Order Sub-Agent
- 16:39Choosing a Threshold and Why 0.8 Is a Starting Point
- 18:34End-to-End Run: Virtual Environment, Dependencies, Parallel Execution
- 21:48Reading the Results: Scores, Reasons and Why Some Show 'None'
- 24:51Wiring DeepEval Into CI/CD
- 25:22G-Eval for Custom Metrics and Safety Compliance Testing
- 27:54Where Eval Tests Fit in the Test Pyramid - Unit or Integration?
- 29:59Q&A: When Human Judgment Beat DeepEval; Cost of Running Evals at Scale
- 33:09Scaling Human Review Without Recreating the Bottleneck
- 34:42Per-Decision vs Final-Outcome Evaluation and Utterance Coverage
- 37:47Choosing Thresholds Per Metric and Evidence Before Production Autonomy
related talks
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
25mEvaluating An LLM's Ability To Code
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
28mTrustworthy LLMs: Vibe checks are not all you need
32mA Practical Guide to LLM Evaluation
38mMaking LLMs reliable - A practical framework
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications