Understanding Big Data for Software Engineers
summary
Explains why OLTP (transactional) and OLAP (analytical) need different systems, shows concrete solutions: copy-and-query data warehouses and precomputed OLAP cubes (1993) vs file-backed data lakes with distributed compute using MapReduce and Hadoop, then columnar formats like Parquet or ORC that buffer ~10,000 rows, write per-column stripes and footers with statistics to avoid full reads. Argues most firms sit at terabyte scale (not exabyte), predicts lakehouses, self-service data meshes, and using DuckDB or cloud services (Snowflake/Databricks) instead of building everything.
AI-written summary · about
chapters (6)
- 0:00Start
- 0:04Big Data for Software Engineers
- 11:17Data Warehouse
- 24:07Dark Data
- 27:24Rise of the Data Lake
- 47:24The Data Lakehouse and the Cloud
related talks
31mData Warehouses Meet Data Lakes
25mPlatform Engineering for Modern Data Infrastructure
02h 05mBig Data and the Data Lake from the Analyst Perspective
40mTo Warehouse or to Lakehouse? And How Do Various Open Source Project Compare?
11mBreaking Data Silos: Enterprise Data Management
34mData Lakehouse: Who, Why and How
42mContemporary Approaches to Data at Scale
43mBreaking Down Data Warehouses: What to Know Before You Choose