Robust Testing Strategies for AI-Driven Applications
summary
Air Canada chatbot misstates policy and triggers a lawsuit, so testing must prove correctness not just usefulness. Build a golden evaluation dataset of representative inputs, use humans to label and validate LM-as-judge scores, and monitor production to catch surprises. For fast deterministic unit tests record LLM HTTP calls with VCR cassette recording (filter secrets, control recording modes) so CI runs offline, cheap, and non‑flaky while integration tests validate grounding and metrics.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 5:31AI system boundaries
- 11:38AI reliability pillars
- 18:45AI performance evaluation
- 24:52Evaluation datasets
- 31:00Production monitoring
- 33:15Unit tests and mocks
- 38:44Key takeaways and closing remarks
related talks
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
19mTesting LLM-Powered Applications
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
47mCan You Trust Your ChatBot: Techniques for Testing LLM Responses
43mTesting Agentic AI Applications: Beyond Traditional QA
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
37mAI agents testing How to evaluate the unpredictable
46mEnhancing Agent Reliability Through Evaluation-Driven Development