State of the GPU(nion)
summary
Kubernetes supports GPUs via the crude "beta 0" resource since 1.3 but currently requires a special kubelet flag and manual exposure of NVIDIA libraries and nvidia-smi, only recognizes one device per node, and gives no CUDA/driver validation; Clarifai cuts deploys from ~3 hours to seconds using this setup. Work in progress includes a device-discovery PR, NVML-based detection, a docker volume plugin (nvidia-docker/media-docker) for CUDA version matching, and a needed rescheduler to defragment GPUs; sharing GPUs is possible today via unsafe tricks with no enforced accounting.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:18About Clarifai
- 2:07Workloads
- 2:50Neural Networks
- 3:48Clarifai
- 4:44No GPU support
- 5:37Be0 API
- 6:09Limitations
- 6:55Nvidia drivers
- 7:47How to do it
- 8:41Why
- 9:13More devices
- 10:49More details
- 14:39Resource monitoring
- 15:20Sharing GPUs
- 17:11What can you do
- 17:45How to become an incidental contribute
- 18:05How did that happen
- 18:27Advice
- 19:44Be patient
- 21:05Improve tests
- 21:23Documentation
- 21:50Questions
related talks
32mEverything you Need to Know about using GPUs with Kubernetes
30mExplain How Kubernetes Works With GPU Like I’m 5
30mRunning AI Workloads in Containers and Kubernetes
38mImproving GPU Utilization using Kubernetes
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...- 32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers