What did AI just do? Making LLMs Observable with OpenTelemetry
summary
Shows how OpenTelemetry is used to capture full LLM and agent traces so every API call, tool invocation and DB write is recorded with prompts and responses, tokens and cost and span attributes for inputs, outputs and latency. Demonstrates two instrumentation options: adding the OTEL SDK (only latency/availability) or an open‑source instrumenter that emits the full agentic trace. Demo trip planner yields a multi‑turn trace and exposes a date/length hallucination (4 days vs 2) for postmortem or automated eval.
AI-written summary · about
chapters (10)
- 0:00AI agent observability intro
- 1:58Why AI agents need tracing
- 3:23Core LLM observability
- 6:32LLM vs agentic observability
- 7:35OpenTelemetry for AI observability
- 8:59Instrumenting AI apps with OTel
- 10:00Demo: trip planning AI agent
- 13:10Evaluating AI agent responses
- 15:37Multi-turn AI evaluation
- 17:22Q&A
related talks
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
31mDeep Dive To AI Agent Observability
01h 02mObserving AI Applications with OpenLit and OpenTelemetry
32mFrom Traces To Action: Auto-Instrumenting LLMs for Observability...
26mPlug-and-Play Observability: OTel Auto-Instrumentation for LLMs in OpenSearch
30mTutorial: Building an Observable AI Agent With an Open Source Stack
40mOptimizing AI Applications for Production with Observability