~/talk

What Do You Do When NVIDIA GPUs Aren't Available for Inference?

links:none on file
slides:none on file

Claim: far more inference runs without NVIDIA than expected — engineered a small Mixture‑of‑Experts “Trinity mini” to avoid huge clusters while training still used eight H100s for RL rollouts. Uses GGUF models, 8‑bit then lower quantization, and llama.cpp/onslaught local engines; lm‑fit estimates weight+KV fit on laptop or workstation. Speculative decoding (deflash/multi‑token drafts) boosts decode throughput ~3–4× on an 8B model. For cloud scale use LMD + VLM/SGLank orchestration; single‑node VLM or llama.cpp for light traffic.

AI-written summary · about

  1. 0:00The compute scarcity crisis
  2. 3:09Overcoming capacity bottlenecks
  3. 6:34Strategic inference decisions
  4. 9:48Building a local AI stack
  5. 12:27Model formats and quantization
  6. 17:51Optimizing inference performance
  7. 21:06Agentic harnesses and tools
  8. 22:49Cloud scaling for enterprise
  9. 24:52Framework for deployment
  10. 26:38The future of inference
  11. 27:45The case for open source
  12. 30:01Q&A and closing thoughts
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.