Unstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
summary
Demonstrates that unstructured sparsity in attention and Mixture-of-Experts must be made structured to hit GPU tensor cores, introducing a binary sparse block compressed format and a fused STDMM+softmax+SpMM kernel that runs on tensor cores and outperforms prior implementations on full and batched graphs. For MoE, fusing gather/scatter with groupedGEMM and using Triton kernels speeds inference for models like OLMoE (64 experts) and Qwen3 (128 experts), but highly skewed routing still degrades throughput; overall result: practical speedups, plus a call for compilers to automate tiling and scheduling.
AI-written summary · about
chapters (16)
- 0:00Unstructured sparsity in AI
- 1:15Outline & transformer basics
- 3:24Sparse attention patterns
- 6:033S pattern in sparse attention
- 6:24Mixture of experts
- 10:43Hardware-efficient algorithms
- 13:15Sparsity for Tensor Cores
- 14:44Structured unstructured sparsity
- 18:24Efficient MoE computation
- 22:02Kernel composition limits
- 23:52Fused kernels for sparse attention
- 27:31Graph dataset results
- 28:57MoE routing bottlenecks
- 31:33Data layout alignment
- 33:47Parallelism & sparsity
- 35:56Conclusion
related talks
19mOptimizations and abstractions for sparse machine learning
12mSmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...
40mSparse models are fast models: Improving DNN inference performance by over 10X
36mML Efficiency for Large Models: From Data Efficiency to Faster Transformers
19mGeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference..
16mRegister Tiling for Unstructured Sparsity in Neural Network Inference
28mFunctional Sparse Tensor Compilation
22mOptimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...