How Do We Compress Giant AI Models to Run at the Edge?
summary
Cloudflare describes a lossless compression that targets BF16 exponents by applying Huffman coding to the exponent stream, decompressing directly into on‑chip shared memory and feeding reconstructed weights to tensor cores without a main‑memory round trip. The technique, called Unweight, achieves 22% model size reduction, yields bit-exact outputs, saves roughly 3 GB of VRAM on Llama 3.1 8B, open‑sources GPU kernels, and publishes the full research so it runs across a 330+ city GPU network.
AI-written summary · about
chapters (9)
- 0:00Running AI at Cloudflare
- 1:13Deploying large AI models on CDN
- 3:21Two-tier GPU infrastructure
- 4:26Optimizing attention layers with KV caching
- 5:49Quantization to reduce size
- 7:15Memory management with page allocation
- 7:58Hardware affinity for cache reuse
- 9:21Optimizing classic ML models
- 10:02Key takeaways
related talks
30mCompactifAI: Quantum-Inspired AI Model Compression
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
15mDesign Compact Deep Learning Models: Small is the New Big
08mHacks to Make LLM Training Faster
20mFireworks AI: Go from Hype to High - Value AI
41mEscaping the Cloud: High-Performance AI in your Browser
24mAccelerating AI on Edge
36mHigh-Performance Distributed Tensorflow