Challenges with Ultra-low Latency LLM Inference at Scale
summary
Union claims an inference service that outperforms managed offerings — “multiples faster” than SageMaker and up to 5× faster than some APIs — by doing three concrete things: trimming container images so ~80% of image files are unnecessary, using peer-to-peer image pulls and streaming model checkpoints from S3 straight to GPU (RDMA/GPU‑direct style) to avoid disk hops, and splitting work across GPUs with a split prefill/decode pipeline plus a smart router and offloaded KV cache so hot contexts avoid recomputation.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:52Local testing to production
- 4:24Fastest inference engine
- 6:11Optimizing container images
- 8:41Peer-to-peer image caching
- 11:56First token latency increase
- 13:40Caching KV pairs
- 15:03Split prefill and decode
- 17:08Horizontal scaling and routing
- 19:36Global deployment and routing
- 21:42Unified AI platform
- 22:24Future of self-deployed LLMs
related talks
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
24mCost and Performance Optimization of LLM Inferencing.
38mOptimizing LLM Inference for the Rest of Us
19mServerless inferencing: an infrastructure point of view
42mStop Treating LLMs Like REST APIs
20mSecure, private LLMs in your cloud with Anyscale Endpoints
26mEfficiently serving LLMs at scale.
43mLLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale