Gen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme
Talkat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 09:50· Pavilion 7 | Level 7.3 | Room S03·
summary
Pipeline uses AWS SQS triggers and KEDA autoscaling to start embedding jobs with Hugging Face and LangChain, stores vectors in Weaviate vector DB (also tested vv8 / Faiss), and serves answers with a self‑hosted mosaic‑ml 7B LLM. Deployment runs across a 10‑cluster fleet (4 EKS, 6 GKE) with a cluster reporting 5 NVIDIA GPUs. Demo shows a live drug‑research Q&A, day‑0 manifests plus day‑2 lift‑and‑shift migration between EKS and GKE, and automated secret/namespace propagation via a spread policy.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:29Defining fleet management
- 3:02Capabilities of fleet managers
- 6:50GenAI pipeline architecture
- 9:46Data gravity and healthcare
- 17:29Fleet manager benefits
- 19:09Live demo and migration
- 25:51Conclusion
related talks
25mEmpowering AI-Driven Drug Discovery: Overcoming Challenges in Building a ML Platform on Kubernetes
19mSecure your healthcare generative AI workloads on Amazon EKS
36mNavigate the future of AI: Deploy generative models on Amazon EKS
35mCase Study: AI-as-a-Service on Kubernetes at Scale and In Production
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
35mBuilding a Medical AI with Kubernetes and Kubeflow
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
17mDevOps for Private AI: A Scalable Platform Using Kubeflow