Scaling LLMs with Go: Prod Patterns for Handling Millions of AI Requests
summary
Assembled handles about 450M requests/year (≈2 trillion tokens, peaks ~150 req/s) and uses Go for low-latency, concurrent LLM services. Requests are fallback by category so summaries hit fast cheap models while reasoning tasks route to stronger models with timeouts and per-call fallbacks. Runtime runs three parallel goroutines for escalation detection, adversarial checks and RAG retrieval, held by a goroutine gate that discards unsafe outputs; same pattern extends to voice with streaming, filler messages and a unified channel interface.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:28450M LM requests per year in Go
- 3:15Reliability with model fallbacks
- 6:07Real-time safety guardrails
- 12:11Latency and unified interface
- 20:29Go over Python for AI
- 21:51Q&A
related talks
15mAI Trends: Insights
28mNavigating LLM deployment: Tips, tricks, and techniques
43mLLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale
15mAI Trends, and Running LLMs in the Enterprise
28mBuilding AI Applications in Go
28mAnyscale's Unified Platform for LLM Development and Deployment
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
27mScaling using Go