What Do You Do When NVIDIA GPUs Aren't Available for Inference?
summary
Claim: far more inference runs without NVIDIA than expected — engineered a small Mixture‑of‑Experts “Trinity mini” to avoid huge clusters while training still used eight H100s for RL rollouts. Uses GGUF models, 8‑bit then lower quantization, and llama.cpp/onslaught local engines; lm‑fit estimates weight+KV fit on laptop or workstation. Speculative decoding (deflash/multi‑token drafts) boosts decode throughput ~3–4× on an 8B model. For cloud scale use LMD + VLM/SGLank orchestration; single‑node VLM or llama.cpp for light traffic.
AI-written summary · about
chapters (12)
- 0:00The compute scarcity crisis
- 3:09Overcoming capacity bottlenecks
- 6:34Strategic inference decisions
- 9:48Building a local AI stack
- 12:27Model formats and quantization
- 17:51Optimizing inference performance
- 21:06Agentic harnesses and tools
- 22:49Cloud scaling for enterprise
- 24:52Framework for deployment
- 26:38The future of inference
- 27:45The case for open source
- 30:01Q&A and closing thoughts
