Xorbits Inference: Model Serving Made Easy
summary
Xinference claims to simplify LLM serving with concrete features: automatic engine selection across NVIDIA AMD Apple silicon, OpenAI‑compatible API and web UI, integrated support for Hugging Face and GGML, and GPU memory tuning that allocates KV cache to prevent OOMs. It demonstrates speculative decoding with 7B→30B up to 100% speedup and reports a dual‑GPU run nearly doubling throughput and cutting latency by ~80% versus one card. The system manages model downloads, launches instances, and runs at scale.
AI-written summary · about
chapters (9)
- 0:00Xorbits Inference intro
- 1:29Open-source LLMs
- 3:43What Xorbits Inference offers
- 6:56Simpler prompt creation
- 9:05Cross-platform hardware
- 11:00GPU memory efficiency
- 14:40Parallel decoding gains
- 18:57Scaling Xorbits Inference
- 20:46Closing & Q&A
related talks
29mMulti-Accelerator PyTorch Serving With NxD Inference and vLLM
02mInferentia2
26mEfficiently serving LLMs at scale.
30mLLM Inference Arithmetics: the Theory behind Model Serving
24mCost and Performance Optimization of LLM Inferencing.
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
26mAmazingly Fast and Incredibly Scalable Inference...
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)