Which GPU for Local LLMs?
summary
Benchmarks run with llama.cpp compare consumer and datacenter GPUs, showing H100 tops generation at ~300 tokens/s while an RTX 4090 often beats H100 on prompt processing; older cheaper GPUs give best bang-per-buck for generation but newer cards win for prompt latency. Quantization finds a sweet spot around Q4/Q5 quantization: quantized models generate faster but can slow prompt processing and cost quality; flash attention reduces compute-buffer memory and raises throughput. Memory splits into model, context (K/V cache), and compute buffer; concurrency and SLA (example: 10 tokens/s) change the right GPU choice.
AI-written summary · about
chapters (12)
- 0:00Introduction to GPUs for LLMs
- 1:23Key factors for GPU selection
- 3:14Model size and performance impact
- 6:28GPU architecture comparisons
- 7:51Prompt vs token generation performance
- 9:34Quantization and quality trade-offs
- 11:42Backend and driver performance differences
- 13:09Memory usage and concurrency
- 14:36Bigger vs smaller GPUs
- 16:26Cost efficiency and bang per buck
- 17:50Use case-driven GPU choice
- 18:54Q&A session
related talks
10mRunning LLMs locally: Practical LLM Performance on DGX Spark
01h 18mOptimizing LLM Training and Inference Performance on GPUs
35mLocal LLMs on low end hardware - a practical perspective
08mHacks to Make LLM Training Faster
28mServing a Frontier Open Model with Ray & vLLM
15mAI Trends: Insights
27mFaster Than Fast: Networking and Communication Optimizations for Llama 3
16mDecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization