Combo Kernels: Horizontal Fusion Optimizat...
summary
Combo kernels in PyTorch Inductor fuse many small independent Triton launches into a single kernel that uses a PID-based dispatch so one launch replaces N, improving SM utilization and cutting driver overhead. It generates per-subkernel block sizes and a flattened grid dispatch with inline subkernel bodies, autotunes each subkernel, and ships in vLLM for LLM inference. On H100 this yields geomean gains of +7.38% HuggingFace and +5.97% TorchBench. per-subkernel block sizes
AI-written summary · about
related talks
25mMojo + PyTorch: A Simpler, Faster Path To Custom Kernels
28mGenerating State-of-the-Art GEMMs for Heterogeneous Hardware with...
10mMaximizing Kernel Development Productivity Under Performance Constraints
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
18mCompiling Strassen-like Matrix Multiplication Algorithms to Fast CUDA Kernels
32mPanel Discussion: The AI Kernel...
23mBlazing Fast GenAI Inference With Torch.compile
36mDL Compiler Panel Discussion