Scaling DSpark Training Using vLLM, Speculators and Mooncake
summary
A reproducible pipeline trains DSpark drafters by streaming hidden states from a GLM5.2 verifier into a distributed trainer, using vLLM extractors, Mooncake as the transfer layer, and Speculators for DDP training. The run uses 48 GB300 GPUs across 12 nodes with four VLM extractor instances, performs online hidden-state distillation rather than dumping to disk, completes two epochs and achieves a 4.15 final validation acceptance length, showing lossless speculative decoding and significant low-throughput speedups. KV-cache-shaped hidden-state smuggling avoids extra storage overhead.
AI-written summary · about
chapters (12)
- 0:00Introduction and overview
- 0:50What is DSpark and speculative decoding
- 3:21How DSpark improves token prediction
- 5:11Introducing Speculators library
- 6:17Kimik 3D Spark training recipe
- 7:42Training steps and data preparation
- 9:07Hidden states extraction methods
- 11:59The Vera GV300 machine setup
- 13:26Scaling training across nodes
- 14:30Performance bottlenecks and future work
- 15:37Why use VLM for draft serving
- 16:00Closing and resources
related talks
19mNo GPU Left Behind: Scaling Online LLM Training With...
16mKeynote: DBRX: Training Modern LLMs on Databricks
10mRunning LLMs locally: Practical LLM Performance on DGX Spark
26mEfficiently serving LLMs at scale.
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
16mAccelerating Design Space Exploration for LLM Training Systems with Multi-experiment...
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability