Lightning Talk: Building a PyTorch‑native VLLM Plugin for IBM Spyre
summary
IBM Spyre plugin is redesigned to be PyTorch‑native, using torch‑spyre PrivateUse1 device to place tensors and leverage torch.compile; the repo has ~550 commits and ~40 contributors. The team reuses upstream vLLM model, runner and worker code, implements a custom vLLM attention backend and iteratively ports CPU layers to Spire, landing fixes in PyTorch 2.12 for decomposition table switching. A working prototype runs Granite 3.3 8B on Spire and shows device activity and correct responses, with testing and model‑runner integration still needing work.
AI-written summary · about
chapters (12)
- 0:00PyTorch-native VLLM plugin intro
- 1:04VLLM Spire plugin evolution
- 2:08PyTorch and Spire integration benefits
- 3:11Reusing VLLM components
- 3:54Iterative layer integration
- 4:57Mixed-device execution challenges
- 5:40Thread-safe table switching
- 6:02Pluggable layer registration
- 7:08VLLM Spire attention backends
- 7:51Live plugin demo
- 8:33Key takeaways and feedback
- 9:17VLLM future directions
related talks
11mLightning Talk: Lessons from Using Pytorch 2.0 Compile in IBM's Watsonx.AI Inference
35mThe State of vLLM
11mSponsored Session: Lightning Talk: PyTorch and Democratization of AI Accelerators
17mLightning Thunder: Supercharged PyTorch for Modern Hardware
08mSponsored Session: Lightning Talk: PyTorch in Production: Boosting LLM Training and...
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
10mvLLM & Deepspeed Updates
10mLightning Talk: Accelerating LLM Training on Cerebras Wafer-Scale...