Lightning Talk: Accelerating PyTorch Models With Torch.compile's C++ Wrapper Mode
summary
torch.compile replaces the Python wrapper with C++ wrapper mode to cut CPU orchestration overhead, claiming up to 39% speedup over default torch.compile on OSS Huggingface benchmarks while also showing smaller gains on TorchBench and detectible platform-wide improvements (H100, B200, MI300). Kernel generation still uses Triton so the implementation offers lazy Triton compilation (runtime-triggered) or eager compile options, requires a host C++ compiler, and increases compile time ~1.3–2x versus the Python wrapper.
AI-written summary · about
chapters (8)
- 0:00Accelerating PyTorch with C++ wrapper
- 1:47Model bottlenecks in PyTorch
- 2:52Torch.compile C++ wrapper basics
- 5:20C++ wrapper mode challenges
- 8:10Custom ops and CUDA graphs with C++
- 10:17Performance results and benchmarks
- 12:45Trade-offs and compilation time
- 13:27C++ wrapper mode availability
related talks
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
15mPyTorch 2.0: Unlocking the Power of Deep Learning with the Torch Compile API
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
23mBlazing Fast GenAI Inference With Torch.compile
36mDL Compiler Panel Discussion
29mKeynote
32mKeynote: Fireside Chat with Soumith Chintala, Co-Creator of PyTorch & Merve Noyan
15mLightning Talk: AOTInductor: Ahead-of-Time Compilation for PT2 Exported Models