Multi-Accelerator PyTorch Serving With NxD Inference and vLLM
summary
NxD Inference packages PyTorch models into neuron executables and runs alongside TorchInductor CUDA kernels inside a vLLM-based Kubernetes cluster so Llama 8B and Qwen 38B backends serve from L40S GPUs and Inferentia/Trainium in one deployment. The demo shows four backends, continuous traffic shifting, automatic recovery from spot termination via rebalance notifications, measured ~30% cost reduction versus GPU on-demand and up to 70% spot discount, and runtime features like continuous batching and speculative decoding.
AI-written summary · about
chapters (13)
- 0:00Cost-optimized LLM inference
- 2:09Cost models and compute pools
- 4:20Neuron and GPUs for compute
- 5:50Multi-accelerator serving demo
- 8:41Operational tooling comparison
- 10:29Cost analysis and spot trade-offs
- 12:15Resilient design for interruptions
- 14:01Workload simulation and recovery
- 16:31Breaking point of deployments
- 17:57Dynamic scaling with Kada
- 20:32NxD Inference optimization
- 23:45Summary and resources
- 24:51Q&A on GPU compatibility
related talks
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
32mScaling Generative AI Inference with llm-d
10mLightning Talk: Pluggable PyTorch LLM Inference Architecture With VLL...
33mllm-d: Kubernetes Native Distributed Inferencing
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
31mThe Evolution of Multi-GPU Inference in vLLM
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
39mAccelerating Open-Source RL and Agentic Inference with vLLM