Lightning Talk: AOTInductor: Ahead-of-Time Compilation for PT2 Exported Models
summary
AOTInductor ahead-of-time compiles torch.export graphs into a ahead-of-time compiled shared library containing C++ runtime glue and precompiled cuda cubin kernels, turning TorchInductor’s Triton kernels and Python buffer/stream management into C++ launch and memory code. A multi-threaded runtime shares weights and uses CUDA streams for concurrency. Results show over 90% pass rate on one benchmark suite, weaker coverage on TorchBench, and consistent speedups versus eager and the default JIT inductor. Available on PyTorch nightly.
AI-written summary · about
chapters (9)
- 0:00AOTInductor intro
- 1:26Why AOTInductor needed
- 2:30AOTInductor architecture
- 5:41JIT vs AOT codegen
- 8:37Runtime for multi-threaded
- 10:05How to use AOTInductor in practice
- 10:48Performance benchmarks
- 12:16Availability & future
- 12:39Q&A
related talks
07mPyTorch 2.0: TorchInductor
36mDL Compiler Panel Discussion
14mLightning Talk: Accelerating PyTorch Models With Torch.compile's C++ Wrapper Mode
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
28mGenerating State-of-the-Art GEMMs for Heterogeneous Hardware with...
10mLightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...
08mA Halide Backend for TorchInductor
08mLightning Talk: Achieving SOTA GEMM Performance: A CuTeDSL Backend for PyTorch Induc...