Building a Lakehouse data platform using Delta Lake, PySpark, and Trino
summary
Clear claim: a practical lakehouse built on Delta Lake table format plus PySpark and Trino yields a single platform for analytics and ML. The demo shows parquet data plus a Delta transaction log, ACID commits, schema enforcement, merge-based updates, time travel and a Trino connector added in 2022. Architecture uses object storage, a Hive/Glue metastore, raw/intermediate/final zones and vacuum for GDPR deletes; example pipelines run in batch and microbatch streaming and the setup proves viable.
AI-written summary · about
chapters (9)
- 0:00Lakehouse concept
- 1:35Data warehouse limits
- 8:17Perfect platform needs
- 8:37Spark processing
- 13:11Delta Lake storage
- 23:22Trino query engine
- 26:12Lakehouse architecture
- 31:05Lakehouse vision
- 33:14Q&A
related talks
28mLakehouse Architecture with Delta Lake and Databricks
45mBuilding a Lakehouse on the Microsoft Intelligent Data Platform
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake
38mGetting Started with Delta Lake
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
25mBuild open format data lakehouse
37mBuild a Modern Data Platform on AWS
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio