A Brief History of Big Data
summary
Traces the engineering milestones that make modern cloud data stacks: the value tradeoff that single events lose value quickly while aggregates gain value over time, precomputed OLAP cubes that run in batch and historically succeed only about 50/50 on nightly ETL jobs, Google’s MapReduce + GFS papers leading to Hadoop which indexes 1 billion pages and demos on a 4,000-node cluster, the rise of Spark/Presto and Kafka/Storm/Flink (2011), and the move from central data lakes toward decentralized data mesh architectures.
AI-written summary · about
chapters (5)
- 0:00Start
- 22:59Hadoop and rise of the Data Lake
- 36:16Distributed Stream Processing
- 39:36Rise of the Cloud Data Warehouse
- 42:54From Data Lakes to Data Meshes
related talks
52mA Brief History of Data
41mOn Data: Sky above, sand below, peace within
34mTales of Data Architecture Evolution
53mBig Data: from mammoth to elephant
46mThe Evolution of the Open Data Ecosystem
19mDemystifying Big Data
15mThe Future of Data
52mData Warehouse, Data Lake, Data Mesh, Oh My: The History of Data Storage