Beyond the Continuum: The Importance of Quantization in Deep Learning
summary
Quantization shrinks models and speeds inference with concrete gains: weight compression cuts LLaMA‑7B from 25 GB to 6 GB and Dolly from 44 to 11 GB, post‑training int8 quantization pushes YOLO V8 from ~46 FPS (fp32) to ~116 FPS (int8) on CPU with negligible mAP drop, calibration on ~300 images and a ~40s quantize step. Uses OpenVINO and NNCF, supports post‑training, quantization‑aware training and mixed precision toggles.
AI-written summary · about
chapters (13)
- 0:00Quantization importance
- 3:10Quantization helps deployment
- 5:40Quantization mathematically
- 8:09Floating point to integers
- 10:38Quantization methods
- 15:39Quantization aware training
- 17:04OpenVINO and NNCF
- 19:50Post-training quantization
- 21:57Live demo YOLO V8
- 24:51Benchmark quantized vs float
- 27:11Further learning resources
- 28:37Survey and giveaway
- 29:40Q&A
related talks
40mUnlocking Local LLMs with Quantization
01h 14mDistributed and concurrent optimization for machine learning
10mQuantization
30mCompactifAI: Quantum-Inspired AI Model Compression
48mPractical Quantization in Keras
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
14mOcto: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny
38mOptimizing vLLM Performance through Quantization