mxfp8, mxfp4, nvfp4 formats and applications in PyTorch
summary
MicroScaling formats MXFP8, MXFP4 and NVFP4 appear as concrete PyTorch features: MXFP8 stores tensors ~2x smaller and achieves up to 2x speed on matrix multiplies with 1x32 block scaling, MXFP4/NVFP4 give ~4x compression and up to 4x speed (NVFP4 is Nvidia-only with 1x6 inner blocks and an FP32 outer scale). PyTorch adds an E8M0 scale dtype (PyTorch 2.7+) and packed FP4 element types (2.8), a public scaled-MM and low‑precision GEM API, and shows fused kernels are essential (fused scaling ~34x faster); benchmarks on B200 GPUs hit near the theoretical gains for large, compute-bound GEMMs.
AI-written summary · about
chapters (10)
- 0:00Microscaling formats in PyTorch
- 1:49MXFP8, MXFP4, NVFP4 formats
- 5:22New PyTorch dtypes
- 6:45Why scaling is critical
- 9:54Scaling recipes
- 13:45Scale matrix multiplication API
- 14:48Performance benchmarks on B200
- 15:37Performance tips
- 19:06Inference considerations
- 20:14Q&A
related talks
25mFp8 Training From Hopper To Blackwell
10mPyTorch Performance
05mApex
06m⚡ Speedy but Sturdy PyTorch Optimizers
26mExploring HiFloat8: A Tapered Format Complementing the FP8 Ecosystem for Robust Model T...
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
17mLightning Thunder: Supercharged PyTorch for Modern Hardware
25mAI acceleration in POWER10