GeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference..
summary
GeneralSparse is an automated SPMM generator for GPUs that maps diverse pruning patterns (random and magnitude) to an orthogonal memory-division plus reduction design, uses a linear regression cost model and recursive code generator targeting A100, and implements hierarchical reduction algorithms plus single/multi-row and row-split strategies for reuse and load balance. Evaluations on SuiteSparse and pruned LLM weights report up to 20.82× speedup over cuSPARSE on matrices and up to 2.33× end-to-end inference gains.
AI-written summary · about
chapters (6)
- 0:00GeneralSparse for LLM inference
- 2:32Sparse matrix multiplication bottleneck
- 4:40SPMM on GPUs
- 7:34GeneralSparse design
- 15:12Experimental setup & results
- 17:48Related work & conclusion
related talks
40mSparse models are fast models: Improving DNN inference performance by over 10X
19mOptimizations and abstractions for sparse machine learning
29mAre Generative Models The New Sparsity?
16mRegister Tiling for Unstructured Sparsity in Neural Network Inference
25mSPARQ: An Accelerator Architecture for Large Language Models with Joint Sparsity and(…)
23mSparse Computing Drives Energy-Efficient Computation for Artificial Intelligence
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
17mSparseZETA: Intelligent Auto-tuner for Designing High-Performance SpMV Programs