Beyond the Prompt: Evaluating, Testing, and Securing LLM Applications
summary
Shows practical evaluation and security patterns for LLM apps using Vertex AI Evaluation, DeepEval, and Promptfoo with runnable code and SpeakerDeck slides available. Demonstrates that exact-match and BLEU can give misleading scores (BLEU 0.18 on “I am good” vs “I’m good”, ROUGE 66%/83%), uses model-graded judges for trend-based scoring, applies RAG triad metrics (context relevance, groundedness, answer relevance), measures image output with Gecko (two samples average 0.8), and deploys guards like LLM Guard and Model Armor for input/output filtering and red‑team tests.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:30LLM output quality challenges
- 3:34Structured outputs solved
- 5:42Choosing evaluation framework
- 8:59Statistical vs model-based metrics
- 12:50Limitations of deterministic metrics
- 16:23Model-graded metrics reliability
- 25:41RAG and its metrics
- 32:27Evaluating tool use
- 38:01Multimodal LLM with Gecko
- 41:58LLM security frameworks
- 49:23Q&A
related talks
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
32mA Practical Guide to LLM Evaluation
35mEvaluating LLM systems: the nuts and bolts
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
38mMaking LLMs reliable - A practical framework
28mTrustworthy LLMs: Vibe checks are not all you need
25mEvaluating An LLM's Ability To Code