Beyond Static Pods: Dynamic GPU Sharing and Low-Latency Model Switching for LLM ...
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Describes Fast Model Actuation (FMA), an llm-d project that enables dynamic GPU sharing and fast model switching for large-scale Model-as-a-Service on Kubernetes by introducing Dual Pods: a server-requesting Pod that holds GPU quota and a server-providing Pod that runs inference on the allocated GPU. Combines vLLM sleep/wake and a persistent Multi-Instance Launcher to spawn inference subprocesses on demand, reducing model switch time from minutes to seconds and enabling time-division multiplexing of GPUs. Covers the system design, architecture, implementation details and includes a demo.
AI-written summary · about
related talks
15mResource Multiplexing in Tuning and Serving Large Language Models
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
16mLlumnix: Dynamic Scheduling for Large Language Model Serving
32mDynamic Scheduling for Large Language Model Serving
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
34mDeclarative Device Virtualization: Orchestrating GPUs & Hardware in Cloud Native Environments