How to Build Enterprise-Grade AI Agents Using Robust Evaluation
summary
Shows a concrete pipeline: instrumented agents with MLflow 3 and ML tracing (OpenTelemetry), a vector database retriever, and tool calls logged so tokens, latency and API arguments are visible. Demonstrates three evaluation modes — final response, single-step, and trajectory evaluation — and a live run where relevance jumps from ~50% to 100% after tightening the system prompt while product specificity still fails due to a hallucinated feature, exposing where to change prompts, retrievers, or the model.
AI-written summary · about
chapters (12)
- 0:00Welcome
- 4:26LLMs to enterprise AI agents
- 9:25LLMs and RAG limitations
- 11:12Multi-agent systems
- 15:26Customer support AI agents
- 17:54AI project failure rates
- 19:19Evaluating AI agents
- 21:12Measuring agent performance
- 26:12MLflow 3 for GenAI
- 31:15Live demo: agent evaluation
- 37:48Getting started with MLflow 3
- 39:59Q&A
related talks
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
36mAdvanced Evaluation Techniques for AI Agents
33mEvaluating Your AI Agent
41mFrom Lab To Life: Practical AI System Evaluation
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
57mEvaluating AI Agents in production with Python
43mEval-Driven Development: Mastering Agentic Tracing and Expert-Aligned Judges With MLf...