LLMs & Data Warehouses: Reliable AI Agents for Business (Keynote)
summary
LLMs often hallucinate and fail at data aggregation so the talk demonstrates concrete fixes: use semantic models or commented views, feed a DuckDB/MotherDuck SQL dialect markdown file, and keep heavy computation in an analytics engine. Example failures include an AWR (annualized weekly revenue) calculation that dropped prior-month days and intermittent SQL generation; benchmark claim: LLM SQL generation hits roughly 78% SQL accuracy versus ~92% for humans. Production patterns shown include dual execution, zero-copy clones, and per-user duckling instances (read replicas up to 16).
AI-written summary · about
chapters (22)
- 0:00Introduction and agenda
- 1:11LLMs strengths and limitations
- 4:04Why analytics databases matter
- 5:27Row vs column storage explained
- 7:13Tools for database interaction
- 9:23SQL generation accuracy challenges
- 11:10Data sets and context window
- 13:40Fixing calculation errors with context
- 16:59Intermittent LLM behavior issues
- 17:41Semantic modeling for accuracy
- 18:43Simplifying views and comments
- 20:06Preventing runaway queries
- 21:55Defining business metric tools
- 22:17Analytics database capabilities
- 24:05Cost and scaling in data warehouses
- 25:11DuckDB features and benefits
- 26:36MotherDuck architecture overview
- 27:59Dual execution and local compute
- 29:49Zero-copy cloning for agents
- 30:52Scaling with read replica ducklings
- 31:34Real-world agent use cases
- 32:19Q&A session
related talks
38mLLMs for Self-serve Analytics - Lessons Learned
15mFrom Words to Data: Using LLMs to Turn Questions into SQL
32mBeyond LLMs: Building an AI-Powered Decision System at Scale
39mSelf-Service Analytics using LLMs
44mProduction-Grade LLMs & AI Agents
24mWill an LLM replace me? A Systematic study of Gen AI's impact on data jobs
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
24mDemocratizing data access through LLM