AI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
summary
Argues that unit tests fail for probabilistic agents and demonstrates a runnable testbench built on Testkube + Ragas (with DeepEval style verification), Kubernetes and OpenTelemetry that continuously triggers tests on infra changes and publishes metrics to dashboards. Shows a concrete oil finder agent that answers in ~20–25 seconds and requires near 100% correctness for safety, uses an LLM-as-judge with a 90% threshold by default, and ships the agentic layer repo under Apache 2 so it runs end to end in-cluster.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:40Software quality basics
- 1:30Agentic AI challenges
- 3:05Need for AI testing
- 4:37Agent development complexity
- 6:11Testing methodologies
- 19:11Evaluation frameworks
- 21:16Cloud-native test automation
- 25:25Architectural integration
- 27:02Continuous quality loops
- 30:13The Agentic Testbench
- 30:46Standalone evaluation
- 31:14Testbench setup
- 31:24Pipeline workflow execution
- 36:20Ensuring reliability
- 36:38Agentic layer overview
- 37:36Control plane components
- 38:00Summary and conclusion
- 39:23Q&A session
related talks
37mAI agents testing How to evaluate the unpredictable
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
47mBuild Trustworthy AI Agents powered by Evals
43mTesting Agentic AI Applications: Beyond Traditional QA
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
31mObservability in the Matrix: Choosing the Right AI Agent To Escape the Chaos
31mIntroducing Agent Testing
54mYou Can't assertEquals an Agent: A Tester's Guide to Agentic Quality