Low-Precision Inference without Quality Loss...
summary
Claims that FP8 and NVFP4 microscaling cut latency and cost while preserving quality, with NVFP4 yielding ~30% model-size/accuracy efficiency and GEM kernels running ~3.5× faster than FP16 on Blackwell B200. Practical recipe: offline weight quantization, FP8 KV cache often safe, attention is high risk, and runtime activation quantization needs a lightweight pre-quant kernel that adds ~10% overhead. Hardware features required include NVFP4 with 16-value blocks and a quantized global scale, the GEM kernel using TMA and TC G EN05 for pipelined tensor-memory matmuls.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:03Why quantization matters
- 7:04Microscaling formats for FP4
- 11:39Blackwell GPUs and NVFP4
- 14:30Performance gains with NVFP4
- 16:18Quantizing weights and activations
- 17:42Optimizing GEMM kernels
- 21:58TMA and TCG Gen05
- 24:24Tensor memory optimizations
- 25:28Summary
related talks
40mUnlocking Local LLMs with Quantization
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
22mOptimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...
48mPractical Quantization in Keras
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
25mFp8 Training From Hopper To Blackwell
09mOpen models and open compression