Lightning Talk: Accelerating Inference on CPU with Torch.Compile
summary
Torch.compile CPU backend generates C++ fusion kernels, supports fp32, bfloat16 and int8, and runs dynamic-shape models like GPTJ and Llama with AMX on Sapphire Rapids. Optimizations include GEMM fusions, prepacked blocked weights, constant folding/freezing, and a vectorized bfloat16 path; over 94% of generated kernels vectorize and across ~160 models show geometric mean speedups of about 1.4–2.3x (bfloat16) versus eager. Additional work: flash-attention fusion, Python-free C++ wrapper to remove host overhead, and an IPEX staging backend for more custom kernels.
AI-written summary · about
chapters (11)
- 0:00Torch.Compile CPU backend intro
- 1:08Data types and features evolution
- 2:14Performance gains with Torch.Compile
- 2:37Torch.Inductor CPU compilation
- 4:24Vectorization for CPU
- 5:29INT8 quantization support
- 6:12Python-free model export
- 7:21Flash Attention fusion
- 8:06IPA backend for Intel
- 9:10Future plans and upstreaming
- 9:58Q&A
related talks
23mBlazing Fast GenAI Inference With Torch.compile
14mLightning Talk: Accelerating PyTorch Models With Torch.compile's C++ Wrapper Mode
10mLightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
36mDL Compiler Panel Discussion
17mLightning Thunder: Supercharged PyTorch for Modern Hardware
15mPyTorch 2.0: Unlocking the Power of Deep Learning with the Torch Compile API