vLLM & Deepspeed Updates
summary
vLLM presents an inference engine that runs open models on heterogeneous datacenter hardware, integrates PyTorch features like torch.compile and flex attention, and targets support for MVL72 and AMD 355 while aiming to improve accuracy and cluster-scale serving. DeepSpeed delivers concrete systems: zero and 3D parallelism for trillion‑parameter training, super offload on GH200 achieving >50% MFU and 4× faster offload, Zenflow eliminating GPU stalls for ~4× faster post‑training fine‑tuning, compiler‑guided parallelism with ~80% speedups, Arctic long sequence reaching ~500k tokens on one H100 and universal checkpointing plus DeepVME 20× faster checkpoints.
AI-written summary · about
chapters (3)
related talks
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
26mEfficiently serving LLMs at scale.
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
35mThe State of vLLM
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
31mHigh-Performance MPI and Deep Learning on OpenPOWER Platform