SPARQ: An Accelerator Architecture for Large Language Models with Joint Sparsity and(…)
summary
SPARQ is an ASIC-style accelerator that runs large LLM inference natively on floating-point inputs with 4-bit weights and exploits 2:4 semi-structured sparsity using a hybrid MXUF/MXUSI systolic array. MXUSI adds interp buffers and an SPI metadata path to avoid accumulation conflicts and supports float-by-int MACs without dequantization. Measured on real models, a 66B configuration cuts latency ~26%, shows up to 1.53× area and 1.58× energy efficiency, and beats prior work (1.79× area, 5.27× lower EDP), so the hardware prototype and evaluations validate the claimed gains.
AI-written summary · about
chapters (20)
- 0:00SPARQ and on-device LLM inference
- 1:07LLM compression techniques
- 2:54Motivation for SparkQ hardware
- 3:36Limitations of compressed models
- 5:00Hardware for float-int computation
- 6:27SparkQ architecture overview
- 7:55Input stationary dataflow
- 9:21Output correctness with sparsity
- 10:50Data conflicts in sparse matmul
- 11:51Sparse pattern indicator routing
- 12:32Final SparkQ architecture
- 13:15MXU modules efficiency
- 13:36Inference time improvements
- 14:39Area efficiency for large models
- 15:39Energy efficiency with MXUSI
- 16:21Comparison with Figna
- 17:03Key takeaways and performance
- 17:45Q&A hardware config
- 20:41Q&A array size
- 24:41Closing remarks
related talks
23mRunning an Open Source LLM on Open Source Hardware
19mGeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference..
30mAccelerated LLM Inference with Anyscale
21mScaling LLM Inference with vLLM
16mHow fast are LLM inference engines anyway?
17mLLM Inference on RISC-V Embedded CPUs
17mWaferLLM: Large Language Model Inference at Wafer Scale
40mUnlocking Local LLMs with Quantization