AI in My API Gateway ... But Why?
summary
Built an aori LM extension that proxies OpenAI‑compatible calls and can route to OpenAI Mistral or Hugging Face, with retries, circuit breakers, provider fallback to local LMs, per‑API‑key token rate limits and secret vault injection. It adds semantic caching (embedding + vector search) that turns a ~10s generation into a 71ms cache hit, prompt guards that block secrets or toxic text, and wasm “tool call” functions run on the gateway. Claims: centralized observability, lower token spend, and reusable functions and templates for mock data and HTML generation.
AI-written summary · about
chapters (12)
- 0:00AI in API Gateways
- 1:11API Gateway for LLM APIs
- 5:10Unified LLM interface
- 7:40Resilience for LLM APIs
- 8:45Observability and cost for LLM APIs
- 10:55Semantic caching LLM responses
- 13:05Prompt guards for privacy
- 16:00Authentication for LLM APIs
- 17:48Prompt engineering tools
- 18:32Tool calls in LLM APIs
- 21:05Live demos
- 24:01Conclusion
related talks
31mAI APIs: Taming the Wild West with AI Gateways
31mWhen LLMs Hit Production: Why You Need an AI Gateway
30mAI APIs: Taming the Wild West with AI Gateways
33mPowering AI Capabilities with API Management
26mAI Traffic Is Different: Rethinking Proxy Arc...
26mAPIs in the world of Generative AI
30mClean and Controlled AI with API Management
22mLeading the future: AI and APIs, a powerful duo