LLMs for Self-serve Analytics - Lessons Learned
summary
Prototype connects Metabase to a real warehouse and shows that text-to-SQL fails in the wild: context windows, noisy schemas and undocumented temporary tables produce wrong queries and hallucinations. The team uses RAG with embeddings + vector DB, builds an internal chatbot and LLM agents with tool schemas so agents can call search, query, analyze tools, retry on SQL errors, and even loop. They find agents useful but brittle, suggest agents can self-correct and insist on benchmarks and LLM‑friendly tool design (parameter names, helpful error messages, ~20‑tool practical limit).
AI-written summary · about
chapters (18)
- 0:00LLMs in self-serve analytics
- 1:09Lego Movie analogy
- 3:35Text-to-SQL and RAG
- 6:01Demos fail in production
- 8:08LLM limitations and user communication
- 10:17Improving text-to-SQL results
- 13:08Enriching docs for AI search
- 15:15Semantic layers
- 17:43Agent workflows
- 19:28Advantages and pitfalls of agent systems
- 22:39LLM-friendly tools
- 26:35Feedback loops and errors
- 28:42Benchmarking agents
- 30:27Four key lessons
- 32:16EMT and Thomas update
- 32:59Q&A: Embeddings and frameworks
- 35:49Q&A: Agent tool limits
- 37:14Q&A: Agentic RAG tips
related talks
39mSelf-Service Analytics using LLMs
40mLLMs & Data Warehouses: Reliable AI Agents for Business (Keynote)
15mFrom Words to Data: Using LLMs to Turn Questions into SQL
24mDemocratizing data access through LLM
30mWhat Can LLMs Do with Messy Data?
59mWorkshop: Choosing the Best Open Source LLM for Your Application
42mDataEngBytes 2023 - SYD-T1-02 - Mikiko Bazeley
01h 49mAI engineering with open access LLMs that lie, curse, and steal