Interview
summary
Claims that the metadata problem for data lakes is converted into a data problem and handled with Spark so systems can process metadata in parallel, enabling interactive queries even with hundreds of millions of files. Describes a concrete implementation, Delta Lake, used internally and by customers for streaming and batch, and previews Delta pipelines to express ETL as a single data‑flow graph, managing growth from one table to thousands and planned for open source in the next months.
AI-written summary · about
related talks
45mThe Data Lakehouse: A tech talk for everyone who loves data
20mDelta Lake: Turn your Data Lake into a Lakehouse
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
58mBuild reliable data lakes with Delta Lake & Databricks
39mDelta Lake: Diving into Data Lakes Without the Downsides
19mOpen Source Reliability for Data Lakes
22mReliable data engineering made easy
28mLakehouse Architecture with Delta Lake and Databricks