Data Warehouse, Data Lake, Data Mesh, Oh My: The History of Data Storage
summary
Timeline from 1970s SQL to today: relational OLTP systems hit an ETL bottleneck when reports lock transactional tables, so teams build OLAP/data warehouses and experiment to speed reads by remove foreign keys, denormalize for read latency, switch to column stores or NoSQL, and horizontally scale under CAP tradeoffs. Central data lakes solve availability but often turn into a data lake becomes a data swamp; modern solution is a data mesh = distributed ownership with cataloged data products and SLAs, where lakes/warehouses become mesh nodes.
AI-written summary · about
chapters (7)
- 0:00Introduction to data storage
- 1:32The monolithic database era
- 8:47Separating OLTP and OLAP
- 14:46The data warehouse model
- 28:46Transitioning to data lakes
- 36:14Understanding data mesh
- 44:13Methodology vs technology
related talks
31mData Warehouses Meet Data Lakes
36mIntroducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
34mTales of Data Architecture Evolution
48mData Mesh Paradigm Shift in Data Platform Architecture -
41mOn Data: Sky above, sand below, peace within
42mBuilding Tomorrow's Warehouse Today: Modern Data Warehouse Modelling Insights
47mModern Data Architecture: Breaking the Big Data Monolith
34mData Lakehouse: Who, Why and How