AI agents testing How to evaluate the unpredictable
summary
Projects prioritize unit, integration and end‑to‑end tests wired into CI/CD. Prompts and tools are treated as functions; validate formats with JSON schema checks and run LM judge evaluations for safety and quality. Simulate multi‑step sessions, log traces, assert tool calls and memory isolation, and measure routing precision recall and F1. Keep small labeled datasets (minimum 10 queries) per scenario and use open‑source tooling like evidently for session‑level evaluation and automated regression.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:00AI agents testing
- 4:13Software testing hierarchy
- 10:11Unit tests
- 15:05Ground truth vs direct scoring
- 20:01Integration tests
- 27:30Traditional classification task
- 27:56Quality detection task
- 29:37Quality classification task
- 30:29End to end tests
- 31:40Testing with humans
- 33:03What should be tested
- 34:52Practical ideas
- 35:46Invest in data sets
related talks
43mTesting Agentic AI Applications: Beyond Traditional QA
47mBuild Trustworthy AI Agents powered by Evals
33mEvaluating Your AI Agent
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
23mUnit testing LLM Agents
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
36mAdvanced Evaluation Techniques for AI Agents
57mEvaluating AI Agents in production with Python