Keynote: Powering Generative AI with Kubernetes
summary
Claims a reproducible on‑prem GenAI stack that uses Kubernetes to run mistl 7B instruct for inference and a separate text-embedding-inference model, with a Chroma vector DB and Minio object store wired by an event-driven webhook. Demonstrates a three-node testbed with six GPUs (two RTX 4090 per node), models cached on host/NFS, and a RAG pipeline that turns uploaded PDFs into vectors so an airline chatbot immediately reflects V1→V2 policy changes and supports instant rollbacks.
AI-written summary · about
chapters (6)
- 0:00Generative AI adoption
- 5:13Kubernetes for AI workloads
- 9:06GPU testbed for inference
- 12:22RAG chatbot microservices
- 26:35Live chatbot demo
- 30:34Future of Kubernetes AI
related talks
39mPowering Generative AI with Kubernetes: A Cloud Native Approach
09mKeynote: Kubernetes for GenAI: The New Operating System of the AI-First World
15mKeynote: From Platforms to AI Factories. Has Kubernetes Solved It?
10mKeynote: Enabling AI with Cloud Native
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
49mKubernetes and MLOps for Scalable and Reproducible Generative AI
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
36mNavigate the future of AI: Deploy generative models on Amazon EKS