The advent of the open data lake
summary
Ecosystem shifts from Hadoop-style bundled stacks to composable layers that let teams own storage in blob stores and swap engines: columnar files like Parquet plus a table layer (Iceberg) give snapshot isolation and fast partition pushdowns; Arrow provides zero-copy Arrow IPC for in-memory columnar exchange and DataFusion supplies a reusable vectorized engine; catalogs + OpenLineage track versions and provenance; cloud blob storage decouples compute so warehouses (Snowflake/BigQuery) and custom engines share the same data without vendor lock-in.
AI-written summary · about
chapters (8)
- 0:00Introduction to the data lake
- 1:00The legacy of Hadoop and SQL
- 4:45Building efficient query engines
- 9:32Limitations of coupled architecture
- 12:28Cloud-first data warehouses
- 17:34Components of composable systems
- 24:50The modern open data lake
- 28:27Future outlook and Q&A
related talks
22mBig data tools
55mThe Great Data Debate 2022
31mPanel: Open Data Lakes
24mData Mesh and Lakehouse
41mOn Data: Sky above, sand below, peace within
21mUpgrading from the modern data stack to the modern data lake
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
34mData Lakehouse: Who, Why and How