Who Let the Bots Out? A Guide to Evaluating AI Agents
summary
Claims an open source, reproducible evaluation pipeline that splits agent quality into tool use, trajectory (workflow) evaluation, and goal evaluation. Implements a multi-agent sales assistant using LangGraph planner, web search, Snowflake Cortex Search and Cortex Analyst, and charting; uses TrueLens with ChatGPT as the eval provider to score groundedness, answer relevance, plan quality and adherence. Demo produces charts, surfaces failures (wrong tool calls, missing context, occasional hallucination), and shows inline eval coaching improves relevance though groundedness can vary.
AI-written summary · about
chapters (15)
- 0:00Evaluating AI agents
- 1:46Why evaluation matters
- 2:27Sales assistant with AI
- 5:59Multi-agent tools
- 6:41Breaking AI black box
- 9:29True Lens and RAGG triad
- 10:56RAG to agents evaluation
- 12:22Live demo: sales agent
- 13:12True Lens metrics setup
- 16:17Analyzing agent responses
- 18:26Improving agents with feedback
- 19:32Real-time agent coaching
- 21:31Agent performance comparison
- 23:26Key improvements from feedback
- 24:30Why evaluation matters
related talks
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
33mEvaluating Your AI Agent
57mEvaluating AI Agents in production with Python
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
27mHow to Build AI Agents and Keep Your Sanity
40mAI Agents in Practice
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
35mThe Agentic AI Engineer