Beyond the Prompt: Evaluating, Testing, and Securing LLM Applications
summary
Evaluates LLM apps using concrete tools and numbers: uses Vertex AI Evaluation, DeepEval and Promptfoo for metrics, and LLM Guard plus Model Armor for input/output defenses. Shows deterministic metrics failing on short edits (comparing "I'm good" to "I am good" yields BLEU ≈ 0.18 and ROUGE ≈ 0.66; "the cat lay on the mat" vs "the cat sat on the mat" gives BLEU ≈ 0.48 and ROUGE ≈ 0.83). Uses judge LLMs for model‑graded metrics that agree with humans ~80%, and demonstrates anonymize/deanonymize guards that block or rewrite sensitive prompts.
AI-written summary · about
chapters (13)
- 0:00LLM evaluation and testing intro
- 1:30LLM output quality challenges
- 5:23Choosing LLM eval frameworks
- 8:32Statistical metrics for LLM outputs
- 12:23Limitations of deterministic metrics
- 15:28Model-based evaluation metrics
- 19:20Testing with Promptfoo and DeepEval
- 23:37Evaluating RAG pipelines
- 26:05RAG metrics: retriever vs generator
- 30:38Testing tools, agents, multimodal LLMs
- 34:33Securing LLM applications
- 37:45Blocking inputs with LLM Guard
- 38:28Q&A
related talks
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
35mEvaluating LLM systems: the nuts and bolts
32mA Practical Guide to LLM Evaluation
38mMaking LLMs reliable - A practical framework
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
25mEvaluating An LLM's Ability To Code
28mTrustworthy LLMs: Vibe checks are not all you need