You Can't assertEquals an Agent: A Tester's Guide to Agentic Quality
summary
Shows that assertEquals fails for agentic workflows and gives a five‑layer playbook implemented in demo code: a refund bot that logs tool calls and a refund ledger. Tests count model turns to check reasoning / count model turns, detect missing or out‑of‑order tool calls, catch session leaks by running two users, and enforce a recursion limit (8) to avoid burning tokens. Uses Agent Evals, Microsoft Agent Framework and DeepEval; validates outcomes against the backend ledger and uses an LLM judge to flag mismatches (e.g., claimed ₹100 vs actual ₹52).
AI-written summary · about
chapters (35)
- 0:00You Can't assertEquals an Agent
- 1:39Speaker Intro - Gaurav Khurana, Senior Test Consultant, Microsoft
- 2:45Everyone Has Used assertEquals - Why It Breaks Here
- 3:46Testing Systems That Think for Themselves
- 4:48Non-Determinism: The Same Question, a Different Answer
- 5:50A Five-Layer Model You Can Remember in One Sentence
- 6:51How a Traditional Deterministic Booking System Works
- 8:24What Changed: The Model Decides Which Tools to Call, at Runtime
- 10:29The Non-Returnable Biscuit: Would You Approve This Refund?
- 12:00Correct Agent vs Broken Agent - Identical Outputs
- 13:02The Agent That Skipped the Policy Check and Refunded Anyway
- 14:02Why You Cannot Rely on the Response - the Path Is What Matters
- 15:06An Agent Is a Model in a Loop With Tools
- 16:07The Refund Agent Scenario and the Irreversible Step
- 17:38"Real Teams Make Outcomes Outstanding" - the Mnemonic
- 19:09R - Reasoning: Counting How Many Times the Model Was Called
- 21:12T - Tools: Detecting a Required Tool That Was Never Called
- 22:45Checking Tool Call Order, Not Just Presence
- 23:46M - Memory: When Alice's Thread Leaks Into Bob's
- 26:24Always Test With At Least Two Users
- 26:55O - Orchestration: What Happens When a Service Returns 500
- 28:29Handling Failure Gracefully and Watching the Telemetry
- 29:32Setting a Recursion Limit So the Agent Doesn't Burn Tokens Forever
- 32:06Existing Eval Libraries: Agent Evals, Microsoft Agent Framework, DeepEval
- 33:37Orchestration Also Means Junk Tool Data and Multi-Agent Routing
- 37:10O - Outcomes: The Refund It Claimed vs the Refund Ledger
- 39:15Checking the Backend, Not the Agent's Own Report
- 40:46LLM as a Judge for Grounding vs Tone
- 42:18Recapping the Five Layers
- 44:07The "Are You Sure?" Trick for Finding Issues
- 45:22Assert the Path, Not Just the Destination - Test Like Grey Box
- 46:52Q&A: Measuring Consistency Without Forcing Determinism
- 48:54Q&A: How to Check for Drift in Agent Testing
- 50:25Q&A: Comparing Different Agent Responses Using Golden Data Sets
- 52:00Q&A: Why the Execution Sequence Beats the Final Response
related talks
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications- 02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
47mAgentic Testing: Your Skilled Human Tester
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
43mTesting Agentic AI Applications: Beyond Traditional QA
23mBeyond Tests What to Verify in AI Generated Code
37mAI agents testing How to evaluate the unpredictable
47mEvolution of Quality Engineering in Financial Services