Architecting real-time transactional data lake
summary
Shows a working pipeline that writes CDC and streaming events directly from Kinesis/MSK or relational logs into Apache Iceberg on S3 using Amazon Kinesis Data Firehose, claiming exactly-once delivery and automatic Iceberg table creation and schema evolution. Firehose writes Parquet files, uses a merge-read small-file writes pattern for updates/deletes, supports Postgres/MySQL (RDS/Aurora/self-managed), can use AWS Glue or S3 table buckets for metadata, and exposes CloudWatch metrics, supports KMS encryption and Secrets Manager. Configurable 0–900s buffer / 1–128 MB balances latency and file size.
AI-written summary · about
chapters (11)
- 0:00Introduction and setup
- 1:05Session overview and speaker intro
- 3:15Need for real-time transactional data lakes
- 7:07Challenges with traditional data lakes
- 7:49Apache Iceberg for transactional lakes
- 10:18Data movement challenges
- 13:05Amazon Data Firehose solution
- 16:21Routing data to Iceberg tables
- 19:29Handling inserts, updates, deletes
- 23:00Additional Firehose capabilities
- 25:05Resources and closing
related talks
43mManaging a Data Lake on AWS
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
35mStreaming and Real-time Analytics
41mTransactional data lakes for your analytical workloads
34mBest Practices for Implementing a Data Lake in Amazon S3
55mCollecting Big Data at Scale with Amazon Kinesis
53mDefine data streaming architectures & solutions to fit your needs
31mFrom Warehouse to Lakehouse: Data at Rest