Serving a Frontier Open Model with Ray & vLLM
summary
Deploys DeepSeek-V4-Flash on three B200 GPUs in Kubernetes using Ray and vLLM and shows a simpler tensor+expert setup hitting ~2,200 aggregated tokens/sec with faster first-token latency than the model card reference. Data+expert with speculative decoding adds ~300 tokens/sec; a prefill/decode experiment reports ~1,700 tokens/sec but produces garbage outputs. Uses FP4 indexer for the native 1,000,000-token context (needle‑in‑a‑haystack lookup takes ~60s) and highlights GPU topology and cache pressure tradeoffs.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 3:39Model strategies overview
- 7:30Prefill-decode disaggregation
- 14:09Kubernetes cluster setup
- 16:17Ray service configuration
- 18:27Running performance benchmarks
- 21:38Long context performance test
- 23:02Strategy A and B comparison
- 24:49Broken decoding example
- 25:42Ray dashboard and results
- 26:45Key takeaways and summary
related talks
10mRunning LLMs locally: Practical LLM Performance on DGX Spark
01h 45mRun Frontier AI at Home
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
26mEfficiently serving LLMs at scale.
26mAmazingly Fast and Incredibly Scalable Inference...
38mOptimizing LLM Inference for the Rest of Us
01h 18mOptimizing LLM Training and Inference Performance on GPUs
24mLLM Inference Optimization From Simple To Sophisticated