Azure API Management as a GenAI Gateway.
summary
Azure API Management becomes a GenAI gateway that tracks and controls token usage (demo set to 100,000 tokens/day and 500 tokens/min), enforces short‑ and long‑term token quotas, uses managed identity to hide OpenAI keys, and load‑balances with circuit breakers between regional deployments (one PTU ~150,000 TPM, two deployments ≈300,000 TPM) to fail over when limits hit. Policies shown include openai token metric policy and semantic caching with Azure Cache for Redis (demoed load‑balancing and quota enforcement succeeded; semantic caching was described but not demoed).
AI-written summary · about
chapters (8)
- 0:00GenAI Gateway with Azure APIM
- 1:05Why GenAI APIs need management
- 3:30Azure APIM GenAI capabilities
- 8:24Live demo: GenAI Gateway setup
- 13:22Scaling GenAI deployments
- 21:18Monitoring token quotas
- 25:32Unified AI inference API
- 27:43Next steps for production
related talks
43mUnleash the Potential of APIs with Azure API Management
31mLoad Management for AI Models - Managing OpenAI Rate Limits with Request Prioritization
22mWhen AI Agents Start Acting, APIs Become Execution Contracts
17mThe Future of AI Traffic: What's New in Envoy AI Tra...
48mUsing Azure AI Foundry to manage all your Large Language Models
24mInfuse Generative AI in your apps using
51mThe Next Frontier: Azure OpenAI Models and the Arrival of GPT-4
16mDriving innovation through Azure Red Hat OpenShift and Azure OpenAI