Building a Multi-Cluster Privately Hosted LLM Serving Platform on Ku...
summary
Built a multi-cluster, privately hosted LLM serving platform on Kubernetes that uses Istio service mesh for a central control plane and per-customer data planes, KEDA HTTP scaledObjects to do serverless inference, and a custom agent to provision clusters and blob model stores. Cold-starts drop from tens of minutes by using a Spiegel cluster mirror for image layers, S3 CRT multi threaded downloads and a blob cache so 13B weights download under a minute, and runtime adapters via LoRA adapters plus SafeTensor cached weights to serve many fine tuned models from one base model.
AI-written summary · about
chapters (9)
- 0:00Multi-cluster LLM serving intro
- 4:56Kubernetes LLM serving from scratch
- 6:39LLM serving platform challenges
- 9:30Serverless inference for cost/scalability
- 12:17Autoscaling LLM workloads with KEDA
- 15:07Cold start optimization for LLM
- 19:19LLM weight download optimization
- 23:14Dynamic LoRA adapters
- 25:24Q&A
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
42mStop Treating LLMs Like REST APIs
27mYes You Can Run LLMs on Kubernetes
30mLLMs on Autopilot: Running AI Agents on Kubernetes with Open Source Tools
20mInferencing LLMs in production with Kubernetes and KubeFlow
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes