How to build a platform for AI and analytics based on Apache Iceberg
summary
Claim: Apache Iceberg gives ACID transactions on S3 and becomes a universal analytic storage layer that lets diverse engines share one table spec. Implementation: an S3 bucket + Iceberg tables managed by Tabular (REST catalog, RBAC, auto-optimization) connected to Glue, Athena, EMR, Redshift, Kinesis/Kafka/Flink and Lambda. Demo builds a bucket via CloudFormation, creates table alpha, adds Glue and Athena, inserts one row in Athena and sees it immediately in Redshift, with snapshots/time travel and catalog-driven access controls.
AI-written summary · about
chapters (13)
- 0:00Introduction to AI analytics platform
- 1:04What is Apache Iceberg
- 4:31Why Iceberg solves Hadoop issues
- 7:01Modular data architecture benefits
- 9:08Tabular’s platform overview
- 11:54Demo setup and S3 connection
- 14:23Creating a data warehouse
- 16:30Defining and loading a table
- 17:11Connecting compute engines
- 20:05Querying with Athena
- 21:34Integrating with Redshift
- 23:09Inserting and syncing data
- 24:12Final demo results and summary
related talks
50mDesigning a scalable data platform based on Apache Spark/Iceberg
41mTip of the Iceberg
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
49mHow Apache Iceberg Really Works
34mApache Iceberg.
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
21mDataEngBytes 2023 - MEL-T1-05 - Fabio Ramos