Validation and Observability of AI Agents
summary
A prototype mobile app called MyA connects to a multiagent backend using OpenAPI and Crew AI with FastAPI, a vector database for long-term memory, and Phoenix-style observability. It demonstrates tooling that captures spans, sessions, logs, traces, per-call parameters, total cost and cost per call to validate and observe agents, with a demo of 150 traces (~$2.5) and 14 s end-to-end latency; evaluation uses an LM-as-judge plus a custom hallucination score threshold of 0.5 to test logic, detect anomalies, and tune agents before production.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:56Evolution to agents
- 6:55What are agents
- 10:12Single vs multi-agent
- 13:01Agent validation problems
- 15:54MyA demo
- 20:56App observability setup
- 24:15Query analysis hallucinations
- 31:39System quality assessment
- 33:30Own courses development
- 34:13Q&A
- 41:20Agent economics
- 42:25Evaluating solution quality
- 45:17Agent risks and future
related talks
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
28mThe Hard Truth About AI Agents
26mIf your APIs aren’t agent‑ready, your AI agent is just chat: [...]
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
57mEvaluating AI Agents in production with Python
43mTesting Agentic AI Applications: Beyond Traditional QA
40mAI Agents in Practice
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation