Sponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
summary
Arctic Inference is an open‑source vLLM extension that ships specific optimizations: Shift Parallelism (combines tensor and sequence parallelism to match tensor latency while raising throughput and adapt to bursty traffic), Suffix Decoding (suffix‑tree cache that speculates tens to hundreds of tokens at ~5 µs/token, ~200k tokens/sec on one CPU), SwiftKV (half‑input propagation + distilling Q/V <5% params to recover quality) and Arctic Speculator. Benchmarks report up to 3.4x TTFT, 1.7x throughput, 1.75x TPOT and 2.4–4x generation speed on workloads; available as a pip vLLM plugin.
AI-written summary · about
chapters (8)
- 0:00Arctic Inference goals
- 1:49Advanced parallelism
- 7:52Generation latency with speculative decoding
- 11:02Suffix decoding for agentic tasks
- 17:23Demo: speculative decoding speed
- 18:05Swift KV for batch throughput
- 21:41Open source Arctic Inference
- 24:15Q&A on speculative decoding
related talks
28mArctic Inference: Open-Source LLM Inference Optimizations at Snowflake
30mAccelerated LLM Inference with Anyscale
24mLLM Inference Optimization From Simple To Sophisticated
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
30mLLM Inference Arithmetics: the Theory behind Model Serving
21mScaling LLM Inference with vLLM