Testing the Agents That Look You in the Eye
summary
A testing pipeline that joins any joinable WebRTC URL (Zoom, Google Meet, web WhatsApp) without SDKs mimics a human with a fake camera and mic, generates scenarios and 10–15 personas, runs permutations that would otherwise cost ~7.5 hours per scenario, and evaluates results with HyperExecute and TrueGen and an engine scoring 15+ metrics including Mean Opinion Score. Hard wins reported are tuning lip sync and audio RMS to avoid false silence, and handling turn taking; the live demo runs end‑to‑end and produces pass/fail evidence.
AI-written summary · about
chapters (17)
- 0:00The Next Era of Customer-Facing AI: Agents With a Face
- 3:11Speaker Intro - Srinivasan and Sai, Directors of Engineering, TestMu AI
- 3:42From Deterministic Web Testing to Chat, Voice and Video Agents
- 5:15Why Video Agents Are Different: Avatars, Expressions, Sentiment
- 5:46The Scale Problem: One Scenario x Many Personas x Many Runs
- 7:17Automating the Human: Building a Test Candidate, Not a Script
- 8:50Three Layers: Avatar, the Reasoning Brain, and the Browser
- 10:52Memory and Red Teaming in Multi-Round AI Interviews
- 13:26No SDK, No Test Hooks: Testing Any Joinable WebRTC URL
- 15:31Live Demo: Scenario Generation, Personas, Avatars and Test Profiles
- 20:37Watching Two AI Agents Interview Each Other
- 22:09Evaluation: Goal Completion, Validation Criteria, Conversation Flow
- 23:41The Hardest Problems: Lip Sync, Audio RMS and False Silence
- 26:13Turn Taking: Too Eager, Too Patient, and Dead Air
- 28:1415+ Metrics Including Mean Opinion Score and Avatar Presentation
- 33:16What's Next: Native Apps, Noisy Backgrounds, Multi-Person Red Teaming
- 37:00Q&A: Chaos Engineering, Nodding and Attentiveness, Sign Language, Latency
related talks
43mTesting Agentic AI Applications: Beyond Traditional QA
31mIntroducing Agent Testing
56mAI Agents That Don’t Suck: How to Build Ones That Actually Drive Business Value
20mBuilding the next interface - AI personas who feel as natural as interacting with a human
33mDesigning Voice Agents for Real Conversations
23mUnit testing LLM Agents
33mTrust but Verify: Testing Agents in Copilot Studio
17mBuilding Effective Voice Agents