Quantization at the Edge: Making a 4GB Model Run on 1GB RAM
summary
A practical recipe for running large LLMs on tiny ARM boards: offline quantization plus operator fusion, KV-cache trimming, and runtime memory pooling compress a model enough to avoid peak allocation failures. A demo loads a quantized tiny llama 4-bit quantized model on a Raspberry Pi, original FP16 2110 MB → 637.8 MB (3.3×) and runs at ~82% RAM, turning a 4GB -> 1GB feasibility claim into working code with trade-offs: shorter context, slight accuracy loss, and slower inference than GPUs.
AI-written summary · about
chapters (6)
- 0:00Edge LLM challenges
- 1:10Quantization methods fail
- 4:03Memory spikes in inference
- 8:20Four memory reduction techniques
- 11:37Live demo: 4GB on 1GB RAM
- 13:57Trade-offs and applications
related talks
40mUnlocking Local LLMs with Quantization
29mRunning Large Language Models on Laptops...
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
24mAccelerating AI on Edge
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
15mDesign Compact Deep Learning Models: Small is the New Big
26mMemory-Efficient LLM Inference on Edge Devices With NNTrainer