Fit-to-Serve: How a New DRA Capability for Dynamic Device...
summary
Demonstrates a new DRA capability added to Kubernetes (enable feature gate) that exposes consumable capacity and allow multiple allocations so a single GPU can be partitioned by SM percent and memory and allocated to multiple pods; demo uses the DRA example driver and llm-d ideas on one GPU where a tiny model uses 5% SM + 4GB, a medium uses 30% SM, three mediums plus two tinies fill 100%, and a large pod blocks until capacity is freed — the implementation and demo work end-to-end.
AI-written summary · about
chapters (8)
- 0:00Fit-to-Serve DRA intro
- 2:29Scaling distributed AI inference challenges
- 6:45Scaling granularity for GPU resources
- 8:14Kubernetes device plugin limits
- 11:24How DRA works
- 12:47New DRA features: capacity & multi-allocation
- 16:46Live demo: DRA with GPU partitioning
- 23:03Summary & next steps
related talks
29mWhat Can I Get You? An Introduction to Dynamic Resource Allocation
32mLeveraging DRA for Conquering ML Workload Challenges on...
42mGuide Us on Next Steps in Kubernetes Device Manage...
50mDynamic Resource Allocation - the path towards GA
30mKubernetes WG Device Management - GPUs, TPUs, NICs and More With DRA
27mUncharted Waters: Dynamic Resource Allocation for Networking
41mDynamic GPU Provisioning in Kubernetes With Composable Disaggregated...
34mDeclarative Device Virtualization: Orchestrating GPUs & Hardware in Cloud Native Environments