The Noisy Neighbour Inside Your GPU:...
summary
Continuous batching (vLLM/TGI) fuses strangers on the same GPU step so a 4,000‑token reply can inflate everyone’s inter‑token latency; KV cache for a 70B model uses ~1.3 GB per 4,096 tokens and thrashes when VRAM fills. Instrument with DCGM/Prometheus/vLLM metrics and trace TTFT versus inter‑token latency, KV utilization, prefix‑cache hit‑rate, queue depth and batch occupancy. Alert on KV usage >90% with preemptions>0 or queue depth>0 for 2 minutes; fixes (truncate, smaller batch tokens, more KV memory, host CPU reservations) trade latency vs concurrency.
AI-written summary · about
chapters (17)
- 0:00Introduction and motivation
- 1:23LLM inference vs CPU noisy neighbors
- 3:13How AI models generate text
- 6:02KV cache and memory challenges
- 8:09Prefill vs decode bottlenecks
- 9:12Limitations of traditional metrics
- 10:34Batch dynamics and shared GPU issues
- 11:39Essential telemetry for LLM observability
- 12:44GPU-side monitoring metrics
- 13:48Identifying the root cause
- 14:52Cluster-level shared resource issues
- 16:00Demo setup and tools
- 17:13Interpreting monitoring dashboards
- 19:22Analyzing batch mate behavior
- 20:48Trade-offs of potential fixes
- 21:52Key takeaways and best practices
- 22:33Q&A and closing remarks
