End-to-End Observability for LLM Inference: From Token To GPU
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Practical guide to end-to-end observability for large-language-model inference across ingress, an Inference Gateway, inference engines (vLLM/SGLang), model runtimes, and GPUs. Describes unifying metrics and correlating traces for streaming token-level requests using OpenTelemetry GenAI conventions, and demonstrates debugging three incidents (P99 latency spike, busy-but-idle GPU, KV cache OOM) with an open-source toolchain and configs to speed up incident resolution.
AI-written summary · about
related talks
23mObservability for LLMs: Understanding What’s Happening Under the Hood
06mObservability for Large Language Models
24mHow OpenTelemetry Helps Generative AI
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
30mFrom Black Box To Insight: Observability for AI Agents in Production
29mElevating generative AI and LLM observability
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
12mObserving AI: Why Even LLMs Sometimes Need Therapy