Deep Dive To AI Agent Observability
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 15:15· Level 1 | Hall Entrance N10 | Room E·
summary
OpenTelemetry is used to trace prompts and completions, model settings and tool calls, plus vector DB queries and ranked scores so traces become production eval sets for fine‑tuning and A/B testing. The stack shows Langtrace/OpenTelemetry + OpenAI, Langchain/LlamaIndex/Agno, VB8/pgvector, and vendors like Elastic/Datadog; instrumentation is near zero‑code via runtime patching. Key metrics are time to first token and tokens per second. Multi‑agent demos (Agno orchestrating agents with VB8 retrieval) successfully produce unified per‑request traces to debug nondeterminism, latency and cost.
AI-written summary · about
chapters (13)
- 0:00Introduction and background
- 0:37Defining AI agent systems
- 6:00Challenges in agentic systems
- 6:31Observability for AI agents
- 9:50Traditional vs AI stacks
- 10:55Extended roles of observability
- 19:49OTEL semantic conventions
- 21:44Standardization categories
- 23:12Implementation and tools
- 28:40Instrumentation approaches
- 30:06Full-stack observability map
- 30:28Demonstrating trace analysis
- 30:56Conclusion and Q&A
related talks
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
30mTutorial: Building an Observable AI Agent With an Open Source Stack
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
40mOptimizing AI Applications for Production with Observability
27mKeynote: Tracing the Agent's Mind: Extending OpenTelemetry for Deep...
38mAI Monitoring Mastery: Optimize Performance, Ensure Compliance, and Cut Costs
23mObserving the Observers: Bringing OpenTelemetry to Autonomous AI Agents