Powering Generative AI with Kubernetes: A Cloud Native Approach
summary
A working event-driven RAG platform on a single-node Kubernetes GPU cluster demonstrates live PDF updates transforming chatbot answers: a PDF uploaded to MinIO triggers a webhook that chunks text, calls an embeddings server, stores vectors in Chroma, and injects top-K context into Mistral 7B via Text-Generation-Inference. The lab uses two RTX 4090 GPUs, models loaded to VRAM (100% utilization observed), and a V1→V2 baggage-policy edit instantly changes the bot’s response.
AI-written summary · about
chapters (13)
- 0:00Introduction and demo setup
- 1:49Why Kubernetes for generative AI
- 3:37What is RAG and its importance
- 7:52Live event-driven RAG demo
- 11:53Architecture deep dive
- 15:04Step-by-step Kubernetes deployment
- 21:51Configuring the stack
- 23:14Deploying the LLM on Kubernetes
- 29:43Embeddings model setup
- 32:57Vector database and storage
- 36:08Connecting components with webhooks
- 37:24Deploying the chatbot
- 38:47Q&A and conclusion
related talks
32mKeynote: Powering Generative AI with Kubernetes
36mNavigate the future of AI: Deploy generative models on Amazon EKS
09mKeynote: Kubernetes for GenAI: The New Operating System of the AI-First World
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
20mOperating Frontier Intelligence at Scale
15mKeynote: From Platforms to AI Factories. Has Kubernetes Solved It?
30mDesigning a Hybrid AI Platform on Kubernetes