~/talk

Green AI at Scale: Energy-Efficient LLM Serving using vLLM & LLM Compressor

slides:none on file

Shows concrete techniques and measured gains: vLLM uses PagedAttention plus speculative decoding and continuous batching to cut KV cache fragmentation and increase GPU utilization, yielding about 2.7x throughput versus Hugging Face Transformers in the authors' 100-prompt test. The LLM Compressor pipeline runs W8A8 quantization (SmoothQuant + GPTQ) to halve a 110B Llama‑style model from ~220 GB BF16 to ~106 GB, enabling inference on 160 GB of GPU. CodeCarbon is used to track CO₂ by region.

AI-written summary · about

  1. 0:00Introduction to Green AI
  2. 1:12Energy demand in AI
  3. 3:26Inference energy at scale
  4. 4:29Optimizing LLM inference
  5. 7:01KV Cache and memory issues
  6. 8:27vLLM’s Page Attention
  7. 11:01Model optimization techniques
  8. 12:09LLM Compressor overview
  9. 16:32Quantizing models with LLM Compressor
  10. 19:25Demo: Running large models
  11. 22:38What is Green AI?
  12. 24:47Red AI vs Green AI
  13. 26:15Measuring CO₂ footprint
  14. 28:27Code Carbon for emissions
  15. 29:33Sustainable AI techniques
  16. 30:15Conclusion and Q&A
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.