The Data Lakehouse: A tech talk for everyone who loves data
summary
Explains how Delta Lake adds ACID transactions, time travel and upserts to S3 data lakes and how Delta Live Tables lets pipelines be declared (batch or streaming) so Kafka or file drops feed continuously into tables; shows a live Twitter ingestion to S3, a Hugging Face sentiment classifier and a running Delta pipeline that processed ~6,900 tweets incrementally; cites a 100 TB TPC‑DS run with ~3× throughput and ~12× performance‑per‑cost improvements and previews Project Lightspeed for predictable low latencies.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:56Favorite thing about Apache Spark
- 11:24Structured streaming
- 14:36The Data Lakehouse
- 19:36How does it work
- 23:58Does it perform
- 26:19Streaming
- 28:25Data pipelines
- 30:47Data donation project
- 42:30Streaming in Kafka
related talks
20mDelta Lake: Turn your Data Lake into a Lakehouse
38mGetting Started with Delta Lake
21mBuilding Data Integration Services In Real Time
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
22mReliable data engineering made easy
51mHow to Build a Modern Data Lakehouse?
21mWhy the Lakehouse?
58mBuild reliable data lakes with Delta Lake & Databricks