Scaling RAG Applications to serve millions of users
summary
Launch-to-scale timeline and hard numbers: 250,000 monthly active users, over 2 million files uploaded and peaks above 2 million LLM tokens per minute. Architecture uses Next.js frontend, FastAPI backend, S3, a queue with workers, OCR and file converters, Postgres + PGVector (HNSW), and a separate Postgres for app data. Key moves that actually change outcomes are deleting old vectors, partitioning and indexing, chunking requests, retrying function calls, and migrating to Azure OpenAI for regional rate limits.
AI-written summary · about
chapters (11)
- 0:00Scaling RAG intro
- 1:48What Slides is
- 3:33Growth metrics impact
- 6:25Scaling mistakes
- 9:12Technical architecture
- 11:38Scaling RAG challenges
- 16:35Azure OpenAI migration
- 20:29Vector databases essential
- 21:11Postgres with PG Vector
- 28:15Current and future devs
- 30:00Q&A
related talks
01h 35mTutorial: Build AI-Supercharged RAG Apps with a Vector Database
46mAdvanced RAG with Azure AI Search - top questions answered
31mAIOps: Automating and Scaling ML and LLM Workloads in Production
47mRAG at scale: production-ready GenAI apps with Azure AI Search
54mGetting to 300 million requests a month with a one man AI/Dev-Ops team
17mScaling Enterprise-Grade RAG: Lessons from Legal Frontier
26mProduction RAG at Scale with Azure Database for PostgreSQL
47mReal-Time AI with Open Source