Designing Data Infrastructure in the Age of Generative AI
summary
Delta kernel is a Rust library that embeds the Delta protocol into engines so multiple runtimes can coordinate commits via Unity Catalog instead of per-format logs, enabling cross-format snapshots and multi-table transactions across Apache Iceberg, Delta Lake and others. The talk demonstrates concrete tools—Lance DB and Daft for multimodal storage and GPU-accelerated runtimes, an Iceberg v4 file-format API and a variant type for shredding nested JSON into queryable columns—and reports that catalog-coordinated commits and MLflow-driven agent judges already work end-to-end.
AI-written summary · about
chapters (13)
- 0:00Introduction to data infrastructure
- 1:15Rise of multimodal data
- 3:41Evolving legacy data infrastructure
- 6:30Variant types for semistructured data
- 7:55Blob data management challenges
- 10:00Native multimodal formats and engines
- 11:45Specialization vs modularity in formats
- 13:07Composable data systems explained
- 16:15Delta Kernel for interoperability
- 20:05Unity Catalog as central hub
- 24:12Agent quality tools and MLflow
- 25:56Future of data infrastructure design
- 29:29Q&A on multimodal transactions
related talks
16mThe Evolution of Delta Lake
38mGetting Started with Delta Lake
14mTony Baer, dbinsight & Doug Henschen, Constellation Research | Databricks Data + AI Summit 2023
48mAn Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric
21mBuilding Data Integration Services In Real Time
28mLakehouse Architecture with Delta Lake and Databricks
39mData Observability and Reliability Engineering in a Real-Time World
20mDelta Lake: Turn your Data Lake into a Lakehouse