Generating State-of-the-Art GEMMs for Heterogeneous Hardware with...
summary
Integration of CUTLASS and Composable Kernel into TorchInductor generates and benchmarks thousands of templated GEMM kernels at JIT time, precompiles them, caches artifacts, and selects the fastest per-shape; this yields up to 10% improvement over Triton/cuBLAS on production shapes (typical 6–8% for fused ReLU/FP8, occasional 50–60% max wins), supports torch.compile and AOTInductor, adds dynamic-shape reuse, and uses epilogue visitor trees to fuse pointwise ops successfully.
AI-written summary · about
chapters (6)
- 0:00Motivation and background
- 5:00TorchInductor integration
- 10:39Epilogue fusion techniques
- 19:09Accelerating float8 GEMMs
- 22:45Future work with CuteDSL
- 24:37Questions and answers
related talks
08mLightning Talk: Achieving SOTA GEMM Performance: A CuTeDSL Backend for PyTorch Induc...
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
15mLightning Talk: AOTInductor: Ahead-of-Time Compilation for PT2 Exported Models
10mLightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...
13mLightning Talk: Adding Backends for TorchInductor: Case Study with Intel GPU
10mCombo Kernels: Horizontal Fusion Optimizat...
35mCUTLASS: A CUDA C++ Template Library for Accelerating Deep Learning...
08mA Halide Backend for TorchInductor