Big data tools
summary
A data lake is just an object store plus the objects, and object stores deliver thousands of put/get ops per second and roughly 5–8x cheaper storage than block or file systems. Replacing CSVs with parquet replaces CSV gives columnar compression and nested types, while modern table formats (Delta, Apache Iceberg, Apache Hudi) add metadata logs and atomic upserts. Adding data version control creates repository-level commits, branches and reverts so teams get git-like workflows across tables and lakeFS implements these operations without duplicating data.
AI-written summary · about
chapters (9)
- 0:00Modern data lake improvements
- 1:54Basic data lake and use cases
- 3:20Object stores in data lakes
- 10:07Raw object storage limits
- 11:35CSV to Parquet files
- 12:59Modern table formats
- 15:08Data version control for lakes
- 17:13Data versioning benefits
- 20:25Resilient data pipelines
related talks
15mWhat is a Data Lake, Anyway?
33mThe advent of the open data lake
34mData Lakehouse: Who, Why and How
02h 05mBig Data and the Data Lake from the Analyst Perspective
31mData Warehouses Meet Data Lakes
20mDelta Lake: Turn your Data Lake into a Lakehouse
51mHow to Build a Modern Data Lakehouse?
26mModernize your Enterprise Data Warehouse & generate value from data