~/talk

The Noisy Neighbour Inside Your GPU:...

linkedin:sagar-utekar
github:oshi36
slides:none on file

Continuous batching (vLLM/TGI) fuses strangers on the same GPU step so a 4,000‑token reply can inflate everyone’s inter‑token latency; KV cache for a 70B model uses ~1.3 GB per 4,096 tokens and thrashes when VRAM fills. Instrument with DCGM/Prometheus/vLLM metrics and trace TTFT versus inter‑token latency, KV utilization, prefix‑cache hit‑rate, queue depth and batch occupancy. Alert on KV usage >90% with preemptions>0 or queue depth>0 for 2 minutes; fixes (truncate, smaller batch tokens, more KV memory, host CPU reservations) trade latency vs concurrency.

AI-written summary · about

  1. 0:00Introduction and motivation
  2. 1:23LLM inference vs CPU noisy neighbors
  3. 3:13How AI models generate text
  4. 6:02KV cache and memory challenges
  5. 8:09Prefill vs decode bottlenecks
  6. 9:12Limitations of traditional metrics
  7. 10:34Batch dynamics and shared GPU issues
  8. 11:39Essential telemetry for LLM observability
  9. 12:44GPU-side monitoring metrics
  10. 13:48Identifying the root cause
  11. 14:52Cluster-level shared resource issues
  12. 16:00Demo setup and tools
  13. 17:13Interpreting monitoring dashboards
  14. 19:22Analyzing batch mate behavior
  15. 20:48Trade-offs of potential fixes
  16. 21:52Key takeaways and best practices
  17. 22:33Q&A and closing remarks
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.