Everything You Need to Know About Running LLMs Locally
summary
Explains how to run open‑source LLMs locally with concrete stacks: demos use Llama C++ and Olama for local serving, quantized Llama 3.1 at 4‑bit to cut memory by ~3.5x, VLM for GPU memory efficiency and VLM + KServe on Kubernetes to scale models as a service, and Retrieval‑Augmented Generation with a local vector DB for chat over PDFs. Shows IDE integration via an OpenAI‑spec extension for code assistance and container recipes (Podman/Podman Desktop) for audio→text and agentic tool calling.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:50Running Local LLMs
- 5:40What Youll Learn
- 6:30Why run LLMs
- 8:20Open Source Technologies
- 12:55Models
- 18:25Data
- 19:19Model Serving
- 24:49Code Assistance
- 32:51Models as a Service
- 38:56Wrap Up
- 42:11Code Optimization
- 44:01What Hardware Should You Have
related talks
47mOwn Your LLM: Running Generative and Agentic AI Locally
40mOpen Source LLMs in the Cloud: Scalable Solutions
32mGet Up and Running with Llamastack to Create AI Applications!
01h 19mBuilding an LLM on your Laptop to...
28mChildren's Guide To LLMs on Kubernetes
40mScalable and Efficient LLM Serving With the VLLM Production Stack
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...