Amazingly Fast and Incredibly Scalable Inference...
summary
NVIDIA shows concrete system and model-level tricks to cut LLM inference cost and raise interactivity: NVFP4 4-bit quantization with 16-value block scaling and FP8 block factors and published pre-quantized checkpoints on Hugging Face; speculative decoding plus an MTP Eagle multi-token predictor to raise acceptance rates; helix parallelism that shards KV cache and reuses GPUs for FFN, claiming up to 2x interactivity for >1M token context; Dynamo’s KB block manager tiering (HBM, host RAM, SSD, network), AI configurator finds configs in <1 minute and planner cut time‑to‑first‑token 80% while using 5% fewer GPUs in a partner test.
AI-written summary · about
chapters (18)
- 0:00Scalable LLM inference
- 1:04TensorRT-LLM architecture
- 3:10FP4 quantization and NVFP4
- 5:15Speculative decoding
- 7:03Distributed inference with expert parallelism
- 8:30Prefill-decode disaggregation
- 9:54Helix parallelism
- 11:59Trade-offs in inference
- 12:20Dynamo system optimizations
- 13:44KB-aware routing
- 15:10Multi-tier KV cache
- 16:56KB cache offloading
- 17:17Production serving features
- 19:31AI configurator
- 20:33Real-time traffic adaptation
- 22:22Kubernetes scheduling
- 24:29Fault tolerance in LLM
- 25:55Closing
related talks
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
01h 18mOptimizing LLM Training and Inference Performance on GPUs
22mOptimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...
13mScaling Generative AI Inference at Trillion-Token Scale
28mServing a Frontier Open Model with Ray & vLLM
26mEfficiently serving LLMs at scale.
20mFireworks AI: Go from Hype to High - Value AI
28mThe AI Tipping Point