Lakeside AI: Bringing Models to Data with Open Lakehouses & Agents)
summary
Claim: move models to data by ingesting streaming and batch sources once into Apache Iceberg and serving many AI uses with Trino-powered SQL and agent hooks. Implements managed ingestion (Kafka‑compatible + S3), schema inference, type evolution, dedupe, CDC, compaction, metadata expiration, dead‑letter + reset‑and‑replay, and lineage/observability. Persists embeddings as array<double> columns and exposes SQL LLM functions and a federated query layer. Adds unified model access, rate‑limiting and monitoring via MCP; internal and customer tests show it runs at large scale.
AI-written summary · about
chapters (8)
- 0:00Introduction and team
- 2:20Enterprise AI vision and challenges
- 5:13Building the lakehouse layer
- 12:23Querying with Trino and AI
- 18:26AI agents and data products
- 22:21Governance for AI models
- 25:35Unified architecture and demo
- 28:04Q&A
related talks
28mStreaming Lakehouse with Kafka, Flink and Iceberg
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
31mBuilding a Data Infrastructure for AI/ML
23mBuilding AI-Driven Real-Time Lakehouse APIs
35mHigh-performance data foundation for AI success
58mAI-Enhanced Big Data: Integrating Private LLMs and Vector Databases
28mScaling the Future: Lakehouse Architecture and OpenSource Collaboration
25mHow to build a platform for AI and analytics based on Apache Iceberg