Lightning Talk: Best Practices for LLM Serving with DRA
Lightningat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 10:25· Pavilion 7 | Level 7.3 | Room S03·
summary
Serving LLMs on A100/H100 shows cluster GPU utilization as low as 20–30% and per-token generation around 50 ms per-token under test loads. Experiments show optimal MIG slice size depends on model and concurrent load (latency holds with smaller slices when concurrency is under ~32), but reconfiguring MIG via the default operator forces evictions. A working demo deploys vLLM on Kubernetes using dynamic MIG creation via DRA to scale instances vertically and improve utilization.
AI-written summary · about
chapters (7)
- 0:00Introduction to LLM serving
- 0:31Challenges in model inference
- 2:01Optimization techniques
- 3:22GPU sharing and utilization
- 5:10Dynamic partitioning needs
- 6:36DRA for dynamic allocation
- 7:46Demo and conclusion
related talks
42mStop Treating LLMs Like REST APIs
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
33mAccelerating the LLM Lifecycle on the Cloud
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
40mScalable and Efficient LLM Serving With the VLLM Production Stack
27mYes You Can Run LLMs on Kubernetes
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d