Cost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
summary
Shows how to run thousands of fine‑tuned Llama models on shared GPU/CPU instances using TorchServe and SageMaker Multi-Model Endpoint with PEFT for parameter‑efficient tuning. MME uses model caching, dynamic load/unload, container sharing and smart placement to keep hot models in memory and cold models on S3, demonstrated by a 10‑model demo and an extrapolated ~75% cost reduction for 100 models versus one‑model endpoints. Continuous batching and mixed CPU/GPU serving are supported.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:43LLM deployment trends
- 2:29Cost-effective LLM challenges
- 3:31Amazon SageMaker inference
- 4:11SageMaker MME deployment issues
- 6:21SageMaker MME features
- 7:26Deploying fine-tuned LLMs
- 8:28How SageMaker MME works
- 10:41Touché Serve integration
- 12:03Demo workflow
- 15:45Running the demo
- 17:22Cost savings with MME
- 18:09Q&A
related talks
27mScaling ML to the next level: Hosting thousands of models on SageMaker
39mTrain ML models at scale with Amazon SageMaker Training
29mTrain billion-parameter models with model parallelism on Amazon SageMaker
44mScaling ML from 0 to millions of users
12mMemory-Efficient AI: How PEFT and PyTorch Enable Accessible LLM Fine-Tuning
40mScaling generative AI workloads with efficient model choice
25mAccelerate NLP training with Amazon SageMaker
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...