Building AI-Driven Real-Time Lakehouse APIs
summary
Claims a working pipeline that turns overnight batch reporting into real-time lakehouse APIs, using Databricks for bronze/silver processing and moving the gold layer into Microsoft Fabric to cut latency and centralize governance. Indexes PDFs and SharePoint plus SQL records with Azure AI Search, builds a RAG layer over OneLake so LLM queries return grounded answers; Databricks adoption cited at 74%. Result: faster alerts (avoiding $100,000 delayed-loss scenarios) and improved query accuracy in production demos.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:50Real-time data importance
- 3:33Batch to real-time APIs
- 5:24Lakehouse benefits
- 7:11AI gap in lakehouses
- 8:38Medallion architecture
- 10:06Microsoft Fabric Databricks
- 11:33AI on lakehouse data
- 13:17Real-time alerts decisions
- 14:23RAG with Azure AI Search
- 16:11Databricks Genie queries
- 16:52Azure AI Search integration
- 17:14Closing and key takeaways
- 17:39Q&A: Lake vs lakehouse
- 18:45Q&A: RAG data accuracy
- 20:10Q&A: Multi-tenancy security
related talks
49mBuild & deploy generative AI apps in days with Databricks Lakehouse AI
47mData analytics for the era of AI
44mWhat's the Best Big Data Architecture for You?
31mLeveraging Azure PaaS for Real-Time Social Media Analysis
52mElevate your data and AI governance with Databricks Data Intelligence Platform
49mGenAI for Business on the Modern Data Stack
21mDataEngBytes 2023 - SYD-T1-06 - Sana Sanai
20mKeynote Review