DataEngBytes 2023 - MEL-T1-05 - Fabio Ramos
summary
Apache Iceberg is presented as an open table format that brings ACID transactions, time travel and hidden partitioning to object storage, using data files plus manifest files, manifest lists and snapshots tracked in an AWS Glue catalog. It supports copy‑on‑write or merge‑on‑read updates, works with Spark, Snowflake and other engines, and requires periodic maintenance—compaction, expiring snapshots and removing orphan files—to actually delete data and meet governance and privacy requirements.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:24Apache Iceberg for data governance
- 3:53Iceberg architecture benefits
- 5:20Open table format SQL
- 7:04ACID transactions time-travel
- 8:50Schema evolution compatibility
- 10:17Iceberg vs Hive comparison
- 11:01Iceberg metadata snapshots
- 13:09Copy-on-write vs merge-on-read
- 15:39Data governance maintenance
- 18:07Why adopt Apache Iceberg
related talks
41mTip of the Iceberg
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
50mDesigning a scalable data platform based on Apache Spark/Iceberg
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
25mHow to build a platform for AI and analytics based on Apache Iceberg
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
52mApache Iceberg - An Architectural Look Under the Covers
34mApache Iceberg.