Register Tiling for Unstructured Sparsity in Neural Network Inference
summary
PATCH produces tile-based unstructured sparsity masks using Gumbel-Softmax mask learning over tile choices and 2:4 patterns, POST-trains models for an overall sparsity target (knobbed per model) and yields pruned weights for 128x128 tiles. STOIC extends Triton with a sparse lowering, inspector and auto-tuner that compresses matrices and selects specialized kernels, integrates with PyTorch, and achieves up to 16% A100 speedup on inference while keeping accuracy closer to dense than plain 2:4.
AI-written summary · about
chapters (13)
- 0:00Unstructured sparsity in AI
- 1:18Sparsity for model inference
- 3:05Limitations of 2:4 sparsity
- 4:54Challenges in unstructured sparsity
- 6:18Tile-based unstructured sparsity
- 8:01PATCH model recipe
- 13:46Tile size and partitioning
- 14:53STOIC compiler
- 18:27Kernel specialization in STOIC
- 20:33Auto-tuning in STOIC
- 21:43PyTorch integration
- 22:24PATCH+STOIC speedup on A100
- 23:27Future directions and Q&A
related talks
16mRegister Tiling for Unstructured Sparsity in Neural Network Inference
19mOptimizations and abstractions for sparse machine learning
40mSparse models are fast models: Improving DNN inference performance by over 10X
18mTC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs
17mA Sparse Iteration Space Transformation Framework for Sparse Tensor Algebra
28mFunctional Sparse Tensor Compilation
24mTiling & Parallel Processing of Large Images
01h 00mThe Shape of Things to Come