Running Your Coding Agent Locally: Lessons from a Real-World Experiment
summary
Builds a local coding agent using Llama CPP and LM Studio with GGUF open weights, running quantized models (example Quen 3.5 122B at Q4_KXL) on consumer hardware. Chooses models by intelligence and coding/agentic indexes, checks fit with ainface.co, and tunes quantization, KV cache, GPU offload and batching. Cloud model generates a dashboard in ~15 minutes; local run needs ~48 minutes planning plus ~60 minutes coding with a few debug cycles and no PDF export, tradeoffs are latency versus cost and context memory.
AI-written summary · about
chapters (14)
- 0:00Running coding agents locally
- 2:24Toolchain for local inference
- 4:11Model selection guide
- 5:39Model quantization basics
- 11:13Evaluating model quality
- 14:02Matching models to hardware
- 16:53Context length impact
- 24:01Configuring Llama CPP
- 31:17Llama CPP with Cloud Code
- 32:44Context management for agents
- 34:31Real-world test comparison
- 39:52Local vs cloud models guide
- 42:47Enterprise scalability
- 43:30Q&A
related talks
47mOwn Your LLM: Running Generative and Agentic AI Locally
20mSelf-hosting AI LLMs - a beginners guide
20mOwn your agent
41mBuilding Your (Local) LLM Second Brain
37mEmpowering Developers with local AI
45mEverything You Need to Know About Running LLMs Locally
29mRunning Large Language Models on Laptops...
01h 35mSave your API Keys for someone else