SwiftEP: Accelerating MoE Inference with Buffer Fusion and TMA Offloading
summary
SwiftEP is an all-to-all library that fuses compute and RDMA buffers and offloads NVLink transfers to the Hopper TMA to remove redundant copies and free SMs. It implements WQE post‑coalescing, per‑channel QPs, CUDA IPC for cross‑GPU TMA, and a pull‑based combine. On 16–32 GPU clusters it reports up to 119.7% higher algorithm bandwidth, 66.7% lower SM occupancy, and 21.2% higher request capacity versus DeepEP.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:45MoE inference hardware challenges
- 3:09SM resource observations
- 5:16Buffer fusion and TMA offloading
- 6:21RDMA buffer fusion
- 8:29QP parallelization
- 9:12In-vLink buffer fusion
- 10:34TMA offloading
- 11:59Implementation details
- 12:42Performance evaluation
- 14:08Conclusion
related talks
14mServing Frontier MoE Models at the Lowest Token Cost
26mEfficiently serving LLMs at scale.
15mResource Multiplexing in Tuning and Serving Large Language Models
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
17mCLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
10mvLLM & Deepspeed Updates
10mWeaver: Efficient Multi-LLM Serving with Attention Offloading
27mStageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference