Hosting AI models with Ray and GKE
summary
Deploys a Stable Diffusion model on GKE using Ray Surf and a KubeRay operator, annotating Python classes as Ray actors so code stays unchanged, then autoscaling the cluster (demo scales to five nodes) and serving images via FastAPI with port forwarding; uses Nvidia L4 GPUs for the demo, provides a GitHub runbook, and reports cost savings versus Vertex AI of about €1,000 over months. The live demo has intermittent connection hiccups but ultimately generates images, proving the setup works.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 4:14AI model deployment challenges
- 7:47Kubernetes for AI deployment
- 10:59Ray for distributed AI
- 17:24Ray with Kubernetes benefits
- 18:28Live demo setup
- 24:09Deploying Stable Diffusion
- 28:49Q&A
related talks
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
19mScaling Frontier AI with Ray on Kubernetes
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
35mOperating a Production-Scale Physical AI Data Platform with Ray
31mNext-Generation Data Science Workflows Using Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
35mScaling Ray Train to 10K Kubernetes Nodes on GKE
38mAI in Kubernetes: How to Get Started?