Thunder: Distribute and Optimize Your PyTorch Models With Zero...
summary
Thunder compiles PyTorch by applying composable, full-graph Python transforms that produce debuggable code and enable Megatron-style sharding, kernel fusion with NVFuser and torch.compile. Demonstrated edits: 109B Llama Scout runs on H200 via materialization-to-meta, 8-bit quantization using NVIDIA TransformerEngine, CUDA graphs to remove scheduling gaps, and profiler-linked IR lines to CUDA kernels. Claims include reproducible 2x speedups on Qwen/Llama on Hopper and Blackwell, a custom parameter-sharding transform, and integration with Lightning AI deployment and training stacks.
AI-written summary · about
chapters (11)
- 0:00Thunder and Lightning AI
- 1:27Thunder’s model optimization
- 4:21Thunder’s compiler design
- 8:36IR for distributed execution
- 12:52Optimizing Llama Scout demo
- 15:07Profiling and CUDA graphs
- 16:54Quantizing Llama Scout to 8-bit
- 19:49Transforms and state dicts
- 20:56Nvidia tensor parallelism
- 22:01Future of Thunder and PyTorch
- 23:13Q&A
related talks
17mLightning Thunder: Supercharged PyTorch for Modern Hardware
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
25mInto Generative AI with PyTorch Lightning 2.0
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
20mPyTorch x Transformers Journey
14mLightning Talk: Every Millisecond Counts: The Fine-tuning Journey of an Ultra-Eff...
36mDL Compiler Panel Discussion
21mThe New Vision of Compiler-Accelerated