Scaling LLMs at Bloomberg Using Envoy AI Gateway: A Hybrid Cloud Approach
summary
Bloomberg converts Envoy and Envoy Gateway into a production GenAI control plane called Envoy AI Gateway that front-ends AWS/GCP/Azure and on‑prem models, serving 100+ application teams with a unified OpenAI‑compatible API, OAuth/JWT auth, credential management, and token-based rate limiting. The gateway performs request/response transforms, dynamic routing across providers and regions, passive health ejection and retries, capacity slicing between high/low priority traffic, and reports latency (p50/p90/p99, time‑to‑first‑token); overhead is millisecond‑level and the system runs in production.
AI-written summary · about
chapters (18)
- 0:00Bloomberg’s AI Gateway project
- 1:02Why Bloomberg needs AI
- 3:09Hybrid cloud for AI workloads
- 4:32Envoy AI Gateway with community
- 5:35AI Gateway in Bloomberg’s architecture
- 8:23Declarative AI routing control plane
- 9:49Unified API across model providers
- 11:12Reliability with health checks and retries
- 13:01Multi-cloud routing for performance
- 14:26Onboarding models and users
- 15:50Traffic and cost optimization
- 16:32Balancing AI traffic priorities
- 18:18Dynamic routing with load tracking
- 20:24Rate limiting for cost control
- 21:08Reserve capacity limits
- 22:34Benchmarking model performance
- 24:39Lessons from production deployment
- 25:42Q&A on latency and model selection
related talks
26mAI in My API Gateway ... But Why?
24mHow Bloomberg Uses AI to Help Speed up Over-the-Counter Trading
31mAI APIs: Taming the Wild West with AI Gateways
16mKeynote: LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency
30mAI APIs: Taming the Wild West with AI Gateways
24mPlatform Engineering and DevEx for Your On-Prem LLM
15mScaling AI-Driven Financial Services: Ethical ML at Scale
12mInside ENVOY: Siemens' Production-Grade Agentic AI Services on