llama.cpp - faster and easier for everyone!
summary
llama.cpp adds speculative decoding to run local inference faster by drafting multiple tokens per decode and then verifying them, using implementations like n-gram, small draft models, MTP/MTB heads, and diffusion-based DFlash / DSpark hybrids. Benchmarks show almost 2× speedup for many models with some configurations reporting ~2.1× speedups versus no speculative decoding. The toolchain installs from llama.app or Hugging Face and runs locally via CLI or llama server.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:57What is speculative decoding
- 4:47Speculative decoding implementations
- 7:40Benchmark results
- 8:21How to use speculative decoding
- 9:23Live demo
- 10:28Closing & Q&A
related talks
51mPractical LLM Inference in Modern Java
47mPractical LLM Inference in Modern Java
51mJlama: A Native Java LLM inference engine
30mAccelerated LLM Inference with Anyscale
24mLLM Inference Optimization From Simple To Sophisticated
11mIntroducing Llama API: A Seamless Way to Revolutionize Your Workflow
30mP2P Distributed LLM Inference Across Browsers With llama.cpp And WebRTC
03mFinetune Llama 4 Faster