Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
summary
Quant-LLM demonstrates that FP6 weight quantization preserves LLM quality while enabling much cheaper serving by building a CUDA kernel suite called TC-FPx that fuses dequantization with tensor core matmul and uses ahead‑of‑time prepacking to fix irregular bit layouts. The system runs LLaMA‑70B on a single A100, cuts memory so one GPU suffices, reduces runtime dequant ops to a few bitwise steps, and yields 1.69×–2.65× normalized throughput vs FP16. Source is open source.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:16Motivation for FP6
- 2:46Benefits of FP6
- 5:41TC-FPX kernel design
- 10:25System optimization
- 14:12Performance evaluation
- 17:39Conclusion
related talks
40mUnlocking Local LLMs with Quantization
38mOptimizing vLLM Performance through Quantization
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models
26mLow-Precision Inference without Quality Loss...
29mRunning Large Language Models on Laptops...
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
08mHacks to Make LLM Training Faster
16mDecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization