Beyond the Prompt: Evaluating, Testing, and Securing LLM Applications
summary
Shows how changing prompts or RAG pipelines requires measurement and gives concrete tools and results: uses Vertex AI Evaluation, DeepEval and Promptfoo to run deterministic and model‑graded metrics, notes model‑graded judges agree with humans ~80% of the time, demos Gemini 2.5 structured JSON outputs, defines the RAG triad (context relevance, groundedness, answer relevance), runs red‑team tests and anonymize/deanonymize flows with LLM Guard, and reports these checks catch tool calls, hallucinations and PII leaks.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:37LLM API simplicity demo
- 4:01Defining LLM output quality
- 7:27LLM evaluation frameworks
- 10:39Deterministic vs model metrics
- 13:33Computational metrics demo
- 16:43Model-graded evaluation
- 24:09Retrieval Augmented Generation and metrics
- 31:34Evaluating LLM tools
- 39:19Multimodal LLM evaluation
- 41:26Securing LLM apps
- 47:53Summary
related talks
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
38mMaking LLMs reliable - A practical framework
29mYour Prompt Is Not a Unit Test — instrumenting LLM apps in production
31mBeyond LLMs and Maturing RAG use cases
34mEvaluating LLM Frameworks
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
01h 39mMastering LLMs: From Prompt Engineering to Agentic AI