Fp8 Training From Hopper To Blackwell
summary
FP8 delivers roughly ~4x speed versus FP32 on modern NVIDIA GPUs but only if tensors are scaled to avoid overflows; scaling is applied right before matmuls and stored as small metadata. Blackwell introduces hardware micro-scaling so very small tiles become viable and descaling is cheap, while Hopper forces workarounds: disable fast accumulation (it uses FP22) and handle contiguous K layout copies that obliterate gains. Meta finds E4M3 preferable to E5M2, stochastic rounding reduces underflows, and PyTorch support lives under torch AO (see torch-titan examples).
AI-written summary · about
chapters (15)
- 0:00FP8 training on Nvidia GPUs
- 1:02FP8 improves compute speed
- 2:28Floating-point in FP8
- 5:53Avoiding overflow and underflow
- 7:40Scaling FP8 for stability
- 9:24FP8 only for matmul inputs
- 10:49FP8 scaling strategies
- 13:56Storing FP8 scaling metadata
- 15:23Delayed scaling risks
- 16:24Rotations to reduce underflow
- 17:28FP8 types E4M3 vs E5M2
- 18:31Hopper GPU pitfalls and fixes
- 19:56Memory transfer with FP8
- 20:59Stochastic rounding for FP8
- 23:03PyTorch FP8 with torch AO
related talks
26mLow-Precision Inference without Quality Loss...
25mmxfp8, mxfp4, nvfp4 formats and applications in PyTorch
26mExploring HiFloat8: A Tapered Format Complementing the FP8 Ecosystem for Robust Model T...
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
08mHacks to Make LLM Training Faster
30mOur Journey With TorchTitan
29mEnabling Lightweight, High-Performance FSDP With NVIDIA GPU
10mPyTorch Performance