Testing Agentic AI Applications: Beyond Traditional QA
summary
Concrete failures, tools, counts and fixes: Voice AI Agent: 20+ deployments where the agent correctly parses “Q3 sales figures” then autonomously appends irrelevant market analysis; Phone Caller Agent: 5,000+ patient interactions that pass unit tests but fail with background noise and slow elderly callers; Chat Agent: 100+ daily conversations dropping multi-session context. Demonstrations show agents looping and even switching roles. Testing response: Behavioural Goal Testing, Probabilistic Validation, Adversarial Scenario Generation and Contextual Journey Simulation using platforms like PipeCat, Gemini live, AssemblyAI and 11labs.
AI-written summary · about
chapters (7)
- 0:00Agentic AI testing challenges
- 10:03Traditional vs agentic AI testing
- 17:31Voice assistant tools
- 19:39Live demo: phone call simulation
- 28:20Metrics for agentic AI conversations
- 32:13Channel-specific testing
- 38:18Q&A
related talks
33mTesting at the Speed of Agents: Salesforce's Evolving Approach to Agentic Testing
31mIntroducing Agent Testing
56mAI Agents That Don’t Suck: How to Build Ones That Actually Drive Business Value
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
30mAgentic QA vs LLM Dev: A Live AI Showdown in the DevOps Arena
37mAI agents testing How to evaluate the unpredictable
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
17mChaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents