Green AI at Scale: Energy-Efficient LLM Serving using vLLM & LLM Compressor
summary
Shows concrete techniques and measured gains: vLLM uses PagedAttention plus speculative decoding and continuous batching to cut KV cache fragmentation and increase GPU utilization, yielding about 2.7x throughput versus Hugging Face Transformers in the authors' 100-prompt test. The LLM Compressor pipeline runs W8A8 quantization (SmoothQuant + GPTQ) to halve a 110B Llama‑style model from ~220 GB BF16 to ~106 GB, enabling inference on 160 GB of GPU. CodeCarbon is used to track CO₂ by region.
AI-written summary · about
chapters (16)
- 0:00Introduction to Green AI
- 1:12Energy demand in AI
- 3:26Inference energy at scale
- 4:29Optimizing LLM inference
- 7:01KV Cache and memory issues
- 8:27vLLM’s Page Attention
- 11:01Model optimization techniques
- 12:09LLM Compressor overview
- 16:32Quantizing models with LLM Compressor
- 19:25Demo: Running large models
- 22:38What is Green AI?
- 24:47Red AI vs Green AI
- 26:15Measuring CO₂ footprint
- 28:27Code Carbon for emissions
- 29:33Sustainable AI techniques
- 30:15Conclusion and Q&A
