What Did My Agent Do? Observability and Accountability for AI Agents
summary
Shows how to get concrete traces of multi-step agent runs by combining OpenTelemetry semantics with the OpenLIT SDK so spans include prompts, responses, token counts and which tools run. Demonstrates passive capture via otel-ebpf to add LLM call metadata without redeploying, then enriches those traces with OpenLIT to show agent/tool spans in Grafana Tempo and Prometheus metrics. Uses Claude eval to analyze long traces automatically, provides a runnable repo and shell script; demo captures network and tool calls and the enriched traces do work.
AI-written summary · about
chapters (13)
- 0:00AI agent observability challenges
- 1:47AI agents need observability now
- 4:38Trace-based observability core
- 6:48From fixed to agentic responses
- 8:11Traces as truth for AI
- 9:14LLM vs agent observability
- 11:04OpenTelemetry for AI observability
- 12:30SDKs for AI observability
- 15:19EBPF for LLM tracing
- 17:07Live demo of observability
- 22:54Evaluating agent workflows
- 26:25AI-powered evaluation loop
- 27:29Q&A
related talks
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
31mDeep Dive To AI Agent Observability
30mTutorial: Building an Observable AI Agent With an Open Source Stack
27mKeynote: Tracing the Agent's Mind: Extending OpenTelemetry for Deep...
01h 02mObserving AI Applications with OpenLit and OpenTelemetry
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
27mWhy Are Your AI’s Decisions Hard To Explain: Trace Every Decis...