Building Reliable AI Agents: An Open Source Approach To Evaluation and Observability
summary
Shows an open source stack (TruLens plus an AgentGPT‑style agent) that instruments agent workflows, captures execution traces and visualizes them in a lightweight dashboard; evaluates faithfulness, tool selection accuracy and answer relevancy with an agent‑GPA rubric; demos a developer assistant using Snowflake Cortex + Llama 3.1 against 3 cases (25% of 800, requests/sec math, internal Python indent rule) and reports concrete improvements — act score rises 0.0→0.7 and goal 0.0→0.8 — with code on GitHub.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:10Testing fails for AI agents
- 7:51Evaluation-driven development
- 10:49Agent GPA framework
- 15:05Demo setup
- 19:24Live Agent GPA demo
- 24:35Feedback loops for performance
- 27:50Observability with TruLens
- 28:35Summary
related talks
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
30mTutorial: Building an Observable AI Agent With an Open Source Stack
33mEvaluating Your AI Agent
57mEvaluating AI Agents in production with Python
27mHow to Build AI Agents and Keep Your Sanity
36mAdvanced Evaluation Techniques for AI Agents