A Data Mesh Of Streams
summary
Shows a concrete 10-step implementation: use Kafka/Confluent Cloud with >100 connectors, Schema Registry and a rolling upgrade window for incompatible schema changes, and ksqlDB materialized views for realtime ETL and queries. Data products are event streams stored in compacted topics to provide full datasets and history; multi-cluster replication (MirrorMaker/Confluent replication) spans regions and clouds. A demo prototype (Adam Bellemare) runs on Confluent Cloud and exposes a searchable catalog, lineage and APIs, proving the approach works end-to-end.
AI-written summary · about
chapters (21)
- 0:00Intro
- 3:17Influences
- 7:15Decentralization
- 8:12Physical vs. Virtual
- 11:09Practical example
- 13:43Alice must define herself as a Data Product
- 14:31Event streams decouple you from the source
- 15:26Areas the governance model should touch
- 16:44Data Products are event streams
- 17:37Optimize for out-of-the-box connectivity
- 20:28Use a Schema Registry
- 23:22Compatible changes are handled for you
- 23:43Incompatible changes require a rolling upg window
- 24:58Attach Metadata to Schemas/Topics
- 25:58Search by Data Product
- 27:28Self-serve Data Platform: Events with Histor
- 29:47A Data Mesh is one logical cluster but often mar ones
- 31:31Lineage is even more important for these la more complex implementations
- 32:25Secure Your Data Mesh
- 33:20Implementing a Stream-Based Data Mesh
- 38:3610 Steps to Implementing a Data Mesh with Str 1. Nominate data owners and create an 6. Catalog Data Products to make them environment where Fixat source works searchable in a central
related talks
39mData Streaming? I don't even know her
01h 07mKafka Streams and KSQLDB
31mBridge to the Cloud
23mData Mesh and Data API
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
26mConnecting distributed data with Apache Kafka
43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...