Flexible OpenSearch Data Management With Apache Iceberg: Data...
summary
Proposes using Apache Iceberg as the staging layer between sources and OpenSearch, demonstrating point‑in‑time loads, cost‑effective snapshot versioning and record‑level deletes. Shows a Debezium→Kafka→Spark pipeline with a Spark merge job that applies CDC into Iceberg, then a second Spark job (or Python via PyIceberg) that upserts into OpenSearch using a document ID. Demos include three snapshots, tracked change log view, dynamic schema evolution and distributed indexing with Spark + OpenSearch Hadoop, all working end‑to‑end.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:24OpenSearch ingestion patterns
- 4:00Staging pattern challenges
- 7:51Apache Iceberg introduction
- 8:14What is Apache Iceberg
- 9:22Iceberg as staging layer
- 10:07Iceberg table format components
- 13:43Iceberg key features
- 17:23Iceberg change log view
- 18:27Iceberg and OpenSearch clients
- 18:48Python Iceberg demo
- 20:57Spark with OpenSearch Hadoop
- 23:47Representative use cases
- 24:08Data versioning demo
- 27:46Change data capture use case with demo
- 33:58Dynamic schema changes demo
- 38:24Summary
related talks
01h 12mHands-on with Apache Iceberg
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
50mDesigning a scalable data platform based on Apache Spark/Iceberg
41mTip of the Iceberg
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
39mIntroducing Apache Iceberg: Tables Designed for Object Stores- 45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming