Optimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...
summary
Describes moving MoE weights to NVFP4 to cut memory and keep accuracy on DeepSeek‑V3/R1, with FP4/FP8 KV caching to shrink attention state; NVFP4 uses 16‑element scaling blocks (MXFP4 uses 32). Uses expert placement + replication and Expert Parallelism + Attention Data Parallelism to avoid KV duplication, converts MLA to MQA by weight absorption, and overlaps communication with compute via DualPipe overlapping dispatch/combine while using DeepGEMM, FlashInfer, PDL and CUDA Graphs. NVLink NVL72 provides 1.8 TB/s and rebalancing trims hotspots with only microsecond-level time‑to‑first‑token impact.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:42Mixture of Experts challenges
- 2:27MOE inference strategies
- 3:11Numerical precision with FP8/FP4
- 8:33NVFP4 and MXFP4 formats
- 10:22Quantizing models for FP4 gains
- 11:25Blackwell hardware NVL72
- 13:14Optimizing MOE communication
- 13:58Expert parallelism on NVL72
- 17:53Attention data parallelism
- 21:27Q&A
related talks
26mAmazingly Fast and Incredibly Scalable Inference...
26mLow-Precision Inference without Quality Loss...
19mOffloading-Efficient Sparse AI Systems
26mEfficiently serving LLMs at scale.
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
24mInside MoE Optimization: A Profiler-Guided Tour of torch.compile and vLLM
18mSparse Checkpointing for Fast and Reliable MoE Training
13mOptimizing Dynamic Neural Networks with Brainstorm