DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
summary
DecDEC stores the weight residuals on CPU and dynamically fetches only salient channels (top-K by activation outliers) via zero-copy to correct low-bit errors, keeping quantized weights on GPU. Implementation fuses a dynamic error compensation kernel with base GEMV and uses an auto-tuner for per-layer NTB and K settings. Result: 3-bit Llama-3-8B-Instruct perplexity drops from 10.15 to 9.12, with <0.0003% extra GPU memory and ~1.7% slowdown on an NVIDIA RTX 4050 Mobile; a 2.5% latency target is recommended.
AI-written summary · about
chapters (9)
- 0:00Motivation for on-device LLM inference
- 2:07Challenges of low-bit quantization
- 4:56Channel importance and outliers
- 7:02Dynamic error compensation
- 8:27DecDEC implementation
- 10:41System parameters and tuning
- 13:10Kernel results
- 13:52Accuracy and latency gains
- 15:40Conclusion and release
related talks
38mOptimizing vLLM Performance through Quantization
40mUnlocking Local LLMs with Quantization
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
21mScaling LLM Inference with vLLM
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
01h 14mDistributed and concurrent optimization for machine learning
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...