From Black Box To Insight: Observability for AI Agents in Production
summary
A reference architecture shows traces, metrics and logs collected with OpenTelemetry and visualized in Prometheus and Grafana, deployed via Docker Compose; custom Python detectors flag hallucinations, tool misuse and runaway agents with measured precision improving from 60% to 94%. The demo ties LLM behavior to GPU, container and network metrics, compares GLM4 vs GLM5.2 for cost/performance, simulates batch runs of 50–200 tasks, and calculates GPU cost at $3.50/hr to drive a projected ~40% cost reduction.
AI-written summary · about
chapters (6)
- 0:00AI agent observability intro
- 3:56Agent system challenges
- 7:42Agent hallucination detection
- 11:12Live demo
- 23:45Observability reference architecture
- 28:22Summary & Q&A
related talks
40mOptimizing AI Applications for Production with Observability
21mFrom Black Box to Glass Box: AI-Driven Observability for Modern APIs
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
40mKeep AI on Track: Guardrails + OpenTelemetry Observability
18mHow To Supercharge AI/ML Observability With OpenTelemetry and Fluent Bit
31mDeep Dive To AI Agent Observability
31mBuilding Trustworthy AI: Open Source Monitoring in Action
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures