Register Tiling for Unstructured Sparsity in Neural Network Inference
summary
Introduces Sparse Register Tiling that uses an unroll-and-sparse-jam compile-time transformation plus compression to shrink SpMM code size and improve register reuse; evaluates on 2,396 Transformer and convolution weight matrices (60–95% sparsity) and reports average speedups of 1.72× vs MKL SpMM, 2.65× vs dense GEMM, and 2.12× end-to-end on MobileNetV1 at 70% sparsity on an ARM device; codegen emits ~42 variants for T=8 (≈9–16 for T=4) and integrates into XNNPACK for real gains.
AI-written summary · about
chapters (9)
- 0:00Register tiling for unstructured sparsity
- 1:40Neural network pruning and sparse multiplies
- 3:04Memory hierarchy and tiling
- 4:06Challenges in register-allocated sparse outer products
- 7:15Minimizing code variants with set cover
- 8:41Code generation and SPMM integration
- 10:25Experimental results and gains
- 11:51XNNPACK integration
- 12:12Conclusion and Q&A
related talks
28mRegister Tiling for Unstructured Sparsity in Neural Network Inference
40mSparse models are fast models: Improving DNN inference performance by over 10X
19mGeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference..
19mOptimizations and abstractions for sparse machine learning
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
28mFunctional Sparse Tensor Compilation
17mSparseZETA: Intelligent Auto-tuner for Designing High-Performance SpMV Programs
22mMonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference...