LLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale
summary
Claims a production GenAI stack that streams token-by-token via SSE with FS2 + cats-effect to deliver low-latency responses, routes requests across multiple inference providers with automatic fallbacks and a latch that blacklists failing providers for minutes, and caches and meters usage for multi-tenant fairness. Shows a deployed model family including Balmera 5 1M-token context with sub-22s full-model runs and <300ms function calls, plus type-safe RPC (Poet), Mill builds, and observable, working error-handling in production.
AI-written summary · about
chapters (9)
- 0:00LLMs in the wild intro
- 2:06Enterprise AI challenges
- 7:27Writer’s AI platform
- 11:42Palmera 5 model
- 14:52LLM routing strategy
- 18:26Scala for AI infra
- 27:39Streaming LLMs with SSE
- 31:55Scala tools at Writer
- 38:37Q&A
related talks
28mNavigating LLM deployment: Tips, tricks, and techniques
47mOwn Your LLM: Running Generative and Agentic AI Locally
20mSecure, private LLMs in your cloud with Anyscale Endpoints
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
35mBoost LLMs with Event Streaming and Retrieval Augmented Generation
15mAI Trends, and Running LLMs in the Enterprise
24mScalable GenAI in Production
15mAI Trends: Insights