Local LLMs on low end hardware - a practical perspective
summary
A second‑hand workstation with 32-core, 64GB RAM plus an RTX 3060 12GB GPU costs roughly the same as a Mac mini and runs local LLMs (Qwen 3.5, Gemma 4 via MoE) by offloading unused experts to system RAM. Benchmarks show about 37.5 tokens/sec on the Dell rig versus 12.4 on a Mac mini. Uses: Fedora immutable image for drivers, llama.cpp, llama bench, LM Studio, Ollama, vLLM, PGVector/Qdrant and Hermes agents — it works and is cost effective.
AI-written summary · about
chapters (21)
- 0:00Running LLMs on low-end hardware
- 1:03What this talk aims to provide for attendees
- 2:06Low-end hardware landscape
- 3:50Cloud LLM costs and privacy
- 7:21Defining low-end hardware
- 8:23Second-hand hardware for LLMs
- 9:27DevConf 2025 journey
- 11:13Building hardware from parts
- 14:21Choosing the operating system for stability
- 15:49Cost vs Mac Mini specs
- 17:32Hardware setup trade-offs
- 17:54Inference performance & compatibility
- 19:19Daily driver models
- 20:01Mixture of Experts benefits
- 21:26Benchmarking results
- 23:13Local LLM tools
- 25:00Lessons and pitfalls
- 26:44Model architecture & use cases
- 27:49Self-hosted stack tools
- 29:12LLM home principles
- 30:55Q&A
related talks
47mOwn Your LLM: Running Generative and Agentic AI Locally
23mLLMs on the Edge: The Future of On-Device Intelligence
29mLeveraging open-source LLMs for production
39mMoxin: The All-In-One Tool to Run LLMs on Your Personal Device
45mEverything You Need to Know About Running LLMs Locally
20mSelf-hosting AI LLMs - a beginners guide
52mHW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone