Lightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...
summary
Torch.compile extensions (Inductor + Triton) implement DecomposeK for large-K matmuls and a Custom Op Autotuning pipeline that benchmarks Triton vs ATen, enables epilogue fusion, and adds the first distributed collective-op autotuning. DecomposeK gives up to 28% speedup over ATen with activation fusion (10% without) and fused variants hit 30–40% over cuBLAS; range-based dispatch and per-range autotuning beat Async TP Fused AG+MM by 9% and Async TP Fully Fused by 41% and reach up to 4× in router workloads on V100.
AI-written summary · about
chapters (11)
- 0:00Subgraph fusions and custom op tuning
- 0:21Torch.compile Triton gem backend
- 1:05Split K gemm challenges
- 2:29Decompose K for fusible split K
- 3:32Benchmark decomposed K vs A10/cuBLAS
- 4:56Fusing ReLU with decomposed K
- 5:38DeepSeek V3 MoE router production
- 7:22Future fusion tuning plans
- 8:04Auto tuning distributed collectives
- 8:46Auto heuristic pipelines for dispatch
- 9:28Q&A
related talks
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
23mBlazing Fast GenAI Inference With Torch.compile
16mLightning Talk: Triton Compiler
36mDL Compiler Panel Discussion
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
25mLeveraging AI Ecosystem for Portable and Sustainable GPU Kernels in HPC
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
17mLightning Thunder: Supercharged PyTorch for Modern Hardware