Sparse models are fast models: Improving DNN inference performance by over 10X
summary
Sparse DNNs trade most weights for zeros and use techniques like gradual magnitude pruning, block/partition constraints and complementary sparsity to make matrices hardware-friendly. Open-source toolchains include DeepSparse, TVM and ONNX plus Intel/Microsoft pruning libraries. On current runtimes weight-only sparsity typically yields ~2–4× speedups on CPUs, NVIDIA Ampere’s 50% block constraint gives up to ~2× on GPUs (BERT ~1.5×, ResNet50 ~1.3×), while an FPGA demo using complementary weight+activation sparsity reaches 10–50× (and >100× in best cases) versus a dense 24-core Xeon. Caveats: accuracy can hide subgroup bias and adversarial fragility.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:30Artificial Neural Networks
- 3:20Is that better
- 3:44Hardware challenges
- 5:03The human brain
- 7:16Activation sparsity
- 8:36How much benefit can we get
- 9:31Training a sparse network
- 11:44Gradual magnitude pruning
- 12:32How sparse can I make my networks
- 13:37Examples of sparse networks
- 14:35Challenges with sparse networks
- 16:55Structured sparsity
- 18:20Limiting sparsity
- 18:53Sparse matrices
- 21:34Where are we today
- 23:55Software resources
- 25:09CPU performance
- 26:11ResNet performance
- 27:32Deep Sparse performance
- 28:25GPUs
- 29:59Complementary sparsity
- 31:51Performance demonstration
- 33:09CPU performance comparison
- 34:07Cautions
related talks
23mSparse Computing Drives Energy-Efficient Computation for Artificial Intelligence
19mGeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference..
19mOptimizations and abstractions for sparse machine learning
16mRegister Tiling for Unstructured Sparsity in Neural Network Inference
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
01h 52mNeural Network Accelerator Design
13mFast and Explainable Search on a Budget With OpenSearch Neural Sparse
17mSparseZETA: Intelligent Auto-tuner for Designing High-Performance SpMV Programs