Lightning Talk: Scalable and Observable RAG for Question-Answering in a Box
Lightningat Cloud Native + Kubernetes AI Day Europe 2025 · on Tuesday, 2025-04-01· at 09:45· Level 1 | Hall Entrance N10 | Room G·
summary
Builds a self-hosted RAG stack on Kubernetes using Elotl’s Luna and Nova for autoscaling, VLLM on Ray for distributed inference, and Microsoft 53 plus Rubra AI models for LLMs. Routes queries between a vector DB RAG path and a SQL path via a text-classification random forest that reaches 97% accuracy for the question router. Adds Phoenix (Arize AI) for auto-instrumentation, faces heavy data cleanup and extensive parameter tuning (context length, chunk size, KV cache) to get latency down from dozens of seconds to usable levels.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:04AI infrastructure choices
- 2:51Scaling with VLLM and Ray
- 3:52LLM and agent pipelines
- 4:38RAG and text-to-SQL agents
- 5:40Question router ML
- 6:22Observability with Phoenix
- 7:05Real data role
- 8:07Synthetic data for training
- 8:48Parameter tuning challenges
- 10:38Key lessons
- 11:00Q&A
related talks
41mFrom RAGs to riches: Build an AI document interrogation app in 30 mins
31mOrchestrating Multimodal RAG With Agentic Workflows
36mFrom Theory To Deployment: Building Scalable Multi-Agent Question-Answering Syste...
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
01h 36mTutorial: From Planni...
27mFrom RAGs to riches: Build an AI document interrogation app in 30 mins
30mBuilding a HybridRAG Document Question-Answering System
32mBuilding Performant RAG Applications for Production