Keynote: From Models to Market: What's the Missing Link in Scaling Open Source Models on...
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2023 · on Wednesday, 2023-09-27· at 10:00· 3层 301明珠厅| 3F The Pearl Hall 301·
summary
Open-source models get packaged into a deployable Bento artifact that splits work into an I/O-heavy API server and a compute-heavy Runner, built with BentoML commands and versioned via a Bentofile; OpenLLM provides prepacked LLMs and optimizations (vLLM, TRT backends, continuous batching) so models run with higher throughput. Deployment uses Kubernetes controllers (image-builder and Bento-deployment), Yatai dashboard, and KEDA scale-to-zero with an interceptor/scaler/proxy trio; 95% idle GPU time motivates serverless scaling.
AI-written summary · about
chapters (7)
- 0:00Scaling open source models
- 1:43Model to production challenges
- 3:08BentoML for model packaging
- 6:00Deploying Bentos with Kubernetes
- 8:52Open LLM production
- 12:28Serverless for cost efficiency
- 14:19Summary
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
10mKeynote: Scaling Open Source Large Language Models- 10mKeynote: Scaling Open Source Large Language Models
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
27mThe playbook for a sovereign Model-as-a-Service platform
50mRunning Open LLM Models on Kubernetes
35mServing Large Language Models using KServe
49mOpen Models: The Secret Weapon for Next-Generation Software