Stop Allocating GPUs, Start Delivering Intelligence: An Enterprise...
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
Enterprises face GPUs that cost ~$30,000 each and run at ~15% utilization; moving from charging per-GPU to charging per-token or per-job increases margin. The blueprint uses Kubernetes as the orchestration and a dynamic resource allocation (DRA) device-class model to slice and claim GPU resources, combines vLLM for page‑attention and continuous batching to double token throughput, and adds LMD/semantic routing to cut latency ~47% and token use ~48% while improving accuracy ~10%, with IBM showing ~90% cluster utilization using cloud‑native scheduling.
AI-written summary · about
chapters (13)
- 0:00AI infrastructure management
- 2:11GPU utilization economics
- 5:23Kubernetes as GPU orchestrator
- 8:58Dynamic GPU allocation
- 11:05AI inference with VLM
- 13:52Page attention and memory efficiency
- 15:58Continuous batching for throughput
- 16:41MoE for cost reduction
- 18:08AI gateway vs API gateway
- 19:55Model routing with MoM
- 22:02AI models on Kubernetes
- 27:02Live demo: LMD & KServe
- 34:52Enterprise AI blueprint
related talks
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
28mShared Yet Isolated at Scale: Building Multi-Tenant Inference Platform on…
32mSmart GPU Management: Dynamic Pooling, Sharing, and Scheduling for AI Workloads in Kubernetes
20mHow are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
07mYour GPU delivery is here! Now what? A 5 minute guide to make your CTO happy
30mRun Your Own AI Cluster on a DGX Spark: Kubernetes, GPUs, and DRA