Arctic Inference: Open-Source LLM Inference Optimizations at Snowflake
summary
Arctic Inference is an open‑source vLLM plugin that implements shift parallelism, speculative decoding, suffix decoding and SwiftKV and wires them into vLLM as a pip‑installable plugin. Benchmarks report up to 3.4× faster time‑to‑first‑token, 1.7× higher throughput and 1.75× faster time‑per‑output‑token; generation speedups reach 2.4–4× on agentic workloads and 1.6–1.8× on GPT‑OSS models; SwiftKV doubles input‑token batch throughput and embedding serving sees 4.2–16× gains. Code and papers are available on GitHub.
AI-written summary · about
chapters (8)
- 0:00Arctic Inference intro
- 1:26Trade-offs in LM inference
- 3:14Advanced parallelism
- 8:09Generation latency with speculative decoding
- 14:44Batch throughput with Swift KV
- 18:38Embedding throughput
- 19:39Performance gains and open-source
- 21:44Q&A
related talks
28mSponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
30mAccelerated LLM Inference with Anyscale
24mLLM Inference Optimization From Simple To Sophisticated
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
38mOptimizing LLM Inference for the Rest of Us
39mAccelerating Open-Source RL and Agentic Inference with vLLM
35mThe State of vLLM