An Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric
summary
Claims that Delta Lake adds ACID transactions to data lakes (Databricks released Delta in February 2019 and open‑sourced it the same year), that Lambda→Kappa consolidation becomes feasible when storage and compute support transactions, and that Microsoft Fabric now positions an end‑to‑end analytics stack (Ignite preview adds an OLTP‑style SQL database) drive the talk. Tools named include Databricks, Synapse, Event Hubs, Data Factory and Fabric; attempts to run Spark+Delta on DIY Kubernetes prove fragile so use managed Spark+Delta. Practical patterns include Delta Lake is Spark on acid and treating domains as subscriptions.
AI-written summary · about
chapters (8)
- 0:00Data architecture evolution
- 2:56Data insight problem
- 6:30Delta Lake ACID support
- 16:13Lambda vs Kappa architectures
- 20:33Microsoft Fabric platform
- 26:24Data mesh principles
- 34:33Final thoughts
- 38:24Q&A
related talks
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
38mGetting Started with Delta Lake
28mLakehouse Architecture with Delta Lake and Databricks
51mA Deep Dive into Microsoft Fabric Data Warehouse
19mOpen Source Reliability for Data Lakes
44mDeep-Dive into Delta Lake
55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?
45mBuilding a Lakehouse on the Microsoft Intelligent Data Platform