Observability for LLMs: Understanding What’s Happening Under the Hood
summary
LLM systems require tracing the whole workflow from prompt construction through retrieval, tool calls, model invocations and safety checks, not just 200s and endpoint latency. Real incidents show GPU memory pressure and KV cache growth cause truncation and sudden quality drops, long sessions inflate input tokens and cut concurrency, and empty vector DB hits corrupt answers. Practical stack examples: OpenTelemetry traces, vLLM metrics, DCGM exporter for GPUs, Prometheus + Grafana dashboards tracking time to first token, empty retrieval rate, GPU memory pressure.
AI-written summary · about
chapters (22)
- 0:00LLM observability challenges
- 1:02Traditional monitoring limits for LLMs
- 2:26LLM workflow differences
- 3:29LLM observability definition
- 4:33LLM vs service observability
- 6:37Unpredictability in LLM systems
- 8:03LLM vs classical architecture
- 10:06Four LLM observability layers
- 10:49User signals to track
- 11:29Model and token metrics
- 11:50Retrieval and tool monitoring
- 12:52Infrastructure signals
- 13:33Layer correlation for diagnosis
- 14:38Tracing LLM workflow
- 15:59Why traces matter
- 16:20Safe LLM logging
- 17:02App traces vs infrastructure
- 17:22LLM incident examples
- 19:31LLM metrics and alerts
- 20:14Simple LLM observability stack
- 21:38LLM app health model
- 21:58LLM observability takeaways
related talks
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
15mKeynote: Into the Black Box: Observability in the Age of LLMs
32mEnd-to-End Observability for LLM Inference: From Token To GPU
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
18mBeyond the Chatbot: The Real Impact of LLMs on Observability
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
06mObservability for Large Language Models