Inside BMW's AI Gateway: Self-Hosting Cost-Efficient AI at Enterprise Scale
summary
BMW’s AI Gateway moves from hyperscaler APIs to self-hosting, deploying Hugging Face models with KubeRay with vLLM on AWS EKS and vLLM Omni for diffusion workloads. The platform serves 120+ models across regions, runs GLM 5.2 with tensor/expert parallelism and 4‑bit quantization on GLM 5.2 on 8 RTX Pro cards, and pilots agentic coding for ~100 devs; cost analysis shows a break-even at 2.5B tokens / 16 developers and reports up to 50% savings so far.
AI-written summary · about
chapters (8)
- 0:00Introduction and demo
- 1:55BMW’s AI platform overview
- 5:07AI Gateway vision and evolution
- 8:37Self-hosting with Ray and vLLM
- 11:25Behind the scenes architecture
- 14:13Model serving and hardware setup
- 18:44Cost savings pilot results
- 25:45Results and future outlook
related talks
16mData und AI at Scale: Unlocking Innovation
28mNavigating LLM deployment: Tips, tricks, and techniques
47mEmbedding AI: Don’t Burn Your Fingers in the Heat of the AI revolution
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
36mShift-Left for LLMs: Securing the AI Model Supply Chain
17mUnlocking Germany's AI Potential
45m9 Lessons Learned from Deploying GenAI at Scale