But we have GPUs at home - on-prem ai with kubernetes, Openstack & KubeVirt
summary
Company runs on-prem GPU clusters using bare metal, OpenStack and KubeVirt with the Nvidia GPU Operator; passthrough works but wastes whole cards, vGPU and MIG enable slicing but require expensive vGPU licensing and strict host/guest driver/kernel/CUDA alignment. MMIO limits bite in some OpenStack installers; live migration is generally impractical without InfiniBand. Recommendation: start with KubeVirt, use Ubuntu for easier driver management, expect operational headaches but production inference and retraining can run on-prem with careful planning and dynamic resource allocation coming.
AI-written summary · about
chapters (8)
- 0:00Introduction and speakers
- 1:10Defining GPU platform requirements
- 6:32GPU access methods overview
- 8:17Bare metal and hypervisor approaches
- 11:32GPU sharing with Nvidia vGPU
- 17:52MIG and time slicing methods
- 21:02On-prem feasibility and recommendations
- 28:04Future improvements and Q&A
related talks
11mGPU‑Accelerated Workloads on KubeVirt: Scaling ML/AI in Kuberne...
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
50mMaximizing GPU Utilization for Generative AI
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
37mKubeVirt Deep Dive: Virtualized GPU Workloads on KubeVirt
06mKubernetes Love Machine Learning, Even on Private Cloud
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!