Build Trustworthy AI Agents powered by Evals
summary
Asserts that LGTM is not a deployment strategy and shows concrete eval practices: use deterministic code graders plus LLM-as-judge and human graders, run agents through 800 scenarios (GAIA‑2) in mock app environments, capture full trajectory JSON traces, checkpoint every ~20 steps for 100+ step workflows, force temperature zero and seeds to reduce nondeterminism, and enforce idempotency and saga-style compensation for tool calls. Benchmarks and the open‑source repo let teams catch regressions before users see them.
AI-written summary · about
chapters (13)
- 0:00Why “Looks Good” Is Not Enough for AI Agent Deployment
- 2:44Why Agent Evals Are Harder Than Traditional Software Testing
- 5:46Building Reliable Evals for AI Agent Behavior
- 8:48The Core Layers of an AI Agent Evaluation Framework
- 11:56Testing AI Agents for Hallucinations, Tool Calls, and Non-Determinism
- 12:58Preventing Error Accumulation in Long Agent Workflows
- 16:35Reducing Non-Determinism in AI Agent Testing
- 19:41Making AI Agent Tool Calls Safe and Recoverable
- 23:21Memory Hygiene for Reliable AI Agents
- 29:33Choosing Between Code, Model, and Human Graders
- 37:46GAIA 2: Evaluating AI Agents in Realistic Scenarios
- 42:23Where Frontier AI Models Still Fail
- 43:56What Raw Agent Traces Reveal About Eval Failures
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
37mAI agents testing How to evaluate the unpredictable
36mAdvanced Evaluation Techniques for AI Agents
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
29mBuilding Trust in AI Agents
33mEvaluating Your AI Agent
35mThe Agentic AI Engineer