On Data: Sky above, sand below, peace within
summary
Retrospective across three decades names concrete shifts: data warehouses (Barry Devlin 1988) to data lake (James Dixon, October 2010) to Data Lake House (Snowflake/Databricks, 2020). Describes a live stack using AWS Redshift (migrated nodes for up to 40% performance gain), Apache Airflow running on managed Kubernetes, object storage with Parquet, and Amundsen for discovery. Cites real loads — peaks near 20 million page views per second — and argues for separation of storage/compute, streaming-first pipelines, and governed self-service for data democratization.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:49Data warehouse
- 5:37At the same time...
- 12:38Data Lakehouse
- 20:28Data Science
- 32:58Sand Below
related talks
34mTales of Data Architecture Evolution
51mHow to Build a Modern Data Lakehouse?
43mManaging a Data Lake on AWS
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
52mData Warehouse, Data Lake, Data Mesh, Oh My: The History of Data Storage
33mThe advent of the open data lake
31mData Warehouses Meet Data Lakes
28mYour data lakes need architecture too!