Blazing Fast GenAI Inference With Torch.compile
summary
Torch.compile captures PyTorch graphs with Dynamo and generates fused kernels via inductor, delivering measured speedups like ~60% on Gemma 3, benchmarks from ~1.02x to 2.7x, and vLLM gains around 1.05x–1.9x. Examples show layer norm latency dropping from 1.4 ms to 0.25 ms after removing graph breaks and using dynamic shapes. Workflow details include precompilation CUDAGraphs FlexAttention, regional “compile repeated blocks”, custom inductor passes, backend compile caching (≈60 s for Llama 3 17B) and AOT precompile to avoid warm starts.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 2:51Managing compilation
- 7:25Optimizing Hugging Face Diffusers
- 11:43VLM custom compiler
- 18:06Controlling compilation
- 21:39Closing thoughts
related talks
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
15mPyTorch 2.0: Unlocking the Power of Deep Learning with the Torch Compile API
14mLightning Talk: Accelerating PyTorch Models With Torch.compile's C++ Wrapper Mode
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
27mAccelerating Generative AI
11mLightning Talk: Lessons from Using Pytorch 2.0 Compile in IBM's Watsonx.AI Inference
10mLightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...