Can You Trust Your ChatBot: Techniques for Testing LLM Responses
summary
Builds a Python test planning agent that sends prompts to Gemini, returns structured API test plans, and iterates using human golden datasets and automated scorecards. Demonstrates unit tests and sanity checks for the scaffolding, automated checks for PII and credit card leaks, and prompt‑injection scenarios, showing fixes that reduce but do not eliminate failures because LLM outputs stay non‑deterministic. Provides a GitHub repo, examples, and an AI judge to evaluate and trend quality over time.
AI-written summary · about
chapters (11)
- 0:00How Do You Know If an AI Chatbot Is Actually Correct?
- 3:05Why Are LLM Responses So Hard to Test Reliably?
- 7:14How Does the Funnel Approach Improve LLM Testing?
- 11:25How Do You Test an AI Agent’s Own Responses?
- 16:07What Should You Unit Test Around an LLM?
- 21:49How Do You Define a “Good” LLM Response?
- 22:52How Golden Datasets Improve LLM Evaluation
- 27:35How Do You Test LLMs for PII, Security, and Privacy?
- 33:22How to Automate LLM Quality Scoring and Evaluation
- 35:23How Do You Track LLM Quality as Models Change Over Time?
- 39:29How Do You Test Prompt Injection and Jailbreak Attacks?
related talks
19mTesting LLM-Powered Applications
24mHow to Keep Your LLM Chatbots Real - A Metrics Survival Guide
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
42mRobust Testing Strategies for AI-Driven Applications
17mArchitecting LLM-Based Chatbots using free open-source LLMs
34mEvaluating LLM Frameworks
01h 04mWhat are we thinking in the age of AI?
35mBuilding LLM-based Agents