⚡ Lightning Talk: Unleash the Power of Generative AI Using Only Open Source Technolgies
Lightningat Kubeflow Summit Europe 2024 · on Tuesday, 2024-03-19· at 17:20· Pavilion 7 | Level 7.3 | Room S02·
summary
Deploys open source LLMs on small Kubernetes clusters using KServe + TGIS + Caikit and runs them in production on Red Hat OpenShift. Shows that model sharding via DeepSpeed and KServe torchserve v0.8 let models span GPUs while continuous batching and paged attention increase throughput dramatically. Describes a maintained fork of TGI under Apache 2.0 with production optimizations and Caikit APIs that let application developers use text inputs instead of tensors.
AI-written summary · about
related talks
32mKeynote: Powering Generative AI with Kubernetes
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
10mKeynote: Enabling AI with Cloud Native
09mKeynote: Kubernetes for GenAI: The New Operating System of the AI-First World
36mNavigate the future of AI: Deploy generative models on Amazon EKS
06mSponsored Keynote: Building Your AI Factory: Wh...
43mKeynote: Building AI in the Open: How the Cloud Native Community Shapes What Comes Next