torch.compile Practice and Optimization in Different Scenarios

on YouTube ↗16m

" summary

Practical fixes and passes extend torch.compile (torch.dynamo + autograd + inductor) to real workloads: DeepSpeed + gradient checkpointing compatibility is patched and upstreamed to Hugging Face transformers/accelerate; a pattern-match rewrite for Alibi+mask yields 30% speedup over default fused attention and 260% faster than eager; a constant-fold mask->None unlocks flash attention; padding alignment changes cut recompiles from 7 to 3; work also surfaces startup, dynamic-shape and kernel-caching tradeoffs and ongoing async compile experiments.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to torch.compile
2. 1:44Real-world hurdles
3. 3:12Ecosystem integration
4. 6:02Optimizing kernels
5. 9:48Managing runtime overhead
6. 11:31Future challenges
7. 14:11Conclusion

" related talks

• 15mPyTorch 2.0: Unlocking the Power of Deep Learning with the Torch Compile APIChristian Keller · Open AI + Data Forum @ Open Source Summit North America 2023
• 24mAdvancements in High-Performance AI/ML through PyTorch's Python CompilerSam Gross, Suraj Subramanian, Justin Jeffress · PyCon US 2023
• 06m⚡ Speedy but Sturdy PyTorch OptimizersJane Xu · KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026
• 36mHigh-Performance Distributed TensorflowChris Fregly · PyData London 2017
• 20mGetting Started with Pytorch 2.0 and Hugging Face TransformersPhilipp Schmid · PyTorch Conference 2023
• 31mPyTorch 2.1 - New DevelopmentsSupriya Rao · Scale By the Bay 2023
• 19mIncreasing the Impact of AI Through Better Software (Pytorch Updates)Soumith Chintala · MLconf NYC 2019
• 43mExploring Deep Learning Framework PyTorchStephanie Kim · PyCon 2018