⚡ Speedy but Sturdy PyTorch Optimizers
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
PyTorch reduces memory round trips with horizontal fusion via torch.foreach and vertical fusion via torch.fusedadamw so fewer kernels run; on CUDA 13+ kernels can carry up to ~8x more tensor pointers per launch. A BF16 fused AdamW prototype halves optimizer state memory and benchmarks on a 16B V3 show the same loss and a 25% speedup on the optimizer step. Exposed APIs include torch.optim.functional, CUDA graphs, pre/post-step hooks, and a matrix-based optimizer Muon with a reference and ongoing work on distributed Muon.
AI-written summary · about
related talks
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
10mPyTorch Performance
25mMojo + PyTorch: A Simpler, Faster Path To Custom Kernels
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
16mtorch.compile Practice and Optimization in Different Scenarios
27mAccelerating Generative AI
10mLightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...