Resource-Aware Scheduling for Production GenAI with RAG running on Multicluster Cloud Kubernetes
Talkat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 15:50· Pavilion 7 | Level 7.3 | Room S03·
summary
Claims a reproducible reference architecture that runs GenAI+RAG across multiple Kubernetes clusters using a policy-driven cluster scheduler to place and autoscale components. Uses Nova policy-based scheduler, K8s on EKS, a vector DB (Faiss or Astra/Astro DB), LangChain + FastAPI for RAG, and OpenLLM with a Hugging Face model. Demo runs on six clusters (1 ingestion/vector, 3 CPU query clusters, 2 GPU clusters with one A1G each), supports model rollouts v1→v2→retire and adds v3 via cluster autoscaling, and the deployment works end-to-end.
AI-written summary · about
chapters (15)
- 0:00Resource-aware scheduling intro
- 1:05GenAI needs RAG and self-hosting
- 3:08Reference architecture overview
- 6:05Nova policy-based scheduler
- 7:30Demo components and choices
- 10:44Nova model rollout examples
- 13:32Deploying model v1
- 16:41GPU serving and frontends
- 17:24Vector database population
- 18:26Testing the model
- 19:12Deploying model v2
- 19:56Retiring model v1
- 20:36Autoscaling dynamic needs
- 23:05Summary and try Nova
- 24:08Q&A
related talks
09mKeynote: Kubernetes for GenAI: The New Operating System of the AI-First World
32mKeynote: Powering Generative AI with Kubernetes
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
42mBuilding Scalable RAG Pipelines with Java and Kubernetes
35mCase Study: AI-as-a-Service on Kubernetes at Scale and In Production
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
25mEmpowering AI-Driven Drug Discovery: Overcoming Challenges in Building a ML Platform on Kubernetes
24mScalable GenAI in Production