Self-hosting AI LLMs - a beginners guide
summary
Finds models on Hugging Face (not GitHub), targets about 1–7B parameters for laptops, and uses 4-5 bit quantization to cut model size ~4x with little quality loss for common tasks. Runs work via Olama, Llama.cpp or the LLM CLI to swap local and cloud models; MCP glues agents together. Watch for prompt injection, data exfiltration and hallucinations. Macs with Metal or Nvidia GPUs give big speedups; phones can run tiny models. Demo shows DeepSeek ~1.5B running on a laptop.
AI-written summary · about
chapters (9)
- 0:00Introduction to self-hosting
- 1:08Finding and choosing models
- 3:15Key model terms explained
- 5:42Model evaluation and benchmarks
- 8:30Software for running LLMs
- 11:00Security and hallucination risks
- 13:29Building solutions with MCP
- 15:37Hardware for local LLMs
- 18:27Live demo and Q&A
related talks
47mOwn Your LLM: Running Generative and Agentic AI Locally
01h 35mSave your API Keys for someone else
35mOpen-source AI: why it matters and how to get started
28mNavigating LLM deployment: Tips, tricks, and techniques
47mLaptop-sized ML for Text, with Open Source
31mHugging Face ecosystem for Local AI/ ML
29mRunning Large Language Models on Laptops...
29mLeveraging open-source LLMs for production