Why Your TTFT Lies: Diagnosing PD-Disaggregated LLM Inference With Minimal Cross...
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Describes a unified observability method for production LLM inference in placement‑decoupled (PD) disaggregated architectures, where siloed metrics across Gateway, engines, GPUs and network obscure root causes. Shows how to use a small set of PD‑aware metrics to map observed signals to concrete faults (e.g., prefill queueing, KV cache behavior, KV transfer latency, GPU or network bottlenecks) and to recommend tuning actions like adjusting P:D ratio, batch size, KV policy, and RDMA/topology.
AI-written summary · about
related talks
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
23mObservability for LLMs: Understanding What’s Happening Under the Hood
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
12mBeyond Generic Spans: Distributed Tracing for Actio...
21mBeyond the Theory: What Actually Breaks When You Scale Your Disaggregat...
19mNo GPU Left Behind: Scaling Online LLM Training With...
16mHolmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters