Breakdown New Data Silos in Generative AI
summary
Open-source Gravitino exposes a centralized metadata lake and a federated query engine that pushes operators into Snowflake and Postgres so cross-cloud joins run without full ETL. Demo shows natural language -> text-to-SQL generating queries that join Snowflake (sales) and Postgres (HR) and returns results with a physical execution plan. 0.3 is released with Iceberg support, multi-engine backends (Trino, Spark, Postgres), a roadmap to GA June 2024, and optional materialized caches for hotspots.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:05LF AI & Data Foundation
- 3:16Data backbone of generative AI
- 4:43Data silos challenge
- 6:53Multicloud worsens silos
- 9:26Data stratosphere concept
- 10:53Key features of stratosphere
- 14:28Live demo: federated querying
- 18:05Demo architecture with Gravitino
- 21:39Gravitino unifies metadata
- 24:06Scalable metadata layer design
- 26:35CLI and Spark integration
- 27:59Gravitino milestones
- 29:43Vision for data access
- 30:47Q&A
related talks
31mApache Gravitino: A Multi-regional, Geo-distributed Meta Datalake
50mFrom Transactions to Intelligence: Postgres for Open AI & Analyti
29mBuilding End to End Gen AI Powered Analytics: From Data to Production
50mDesigning a scalable data platform based on Apache Spark/Iceberg
20mOpen Source Database Operations in the Age of AI
32mDesigning Data Infrastructure in the Age of Generative AI
33mStreamlining Metadata Management with PostgreSQL and Apache Gravitino
15mEnterprise AI Runs on Trusted Data