Unifying Real-Time and Data Lake: Yotpo's Transition from Chaos to Coherence
summary
Yotpo replaces costly daily DB snapshots with CDC with Debezium feeding Kafka, builds tables in the lake via streaming, and enforces contracts (Avro + rules) in SDKs so producers cannot break schemas. They add a transaction outbox to guarantee exactly-once event guarantees, integrate Flink, CloudEvents and AsyncAPI, and expose an event catalog in Backstage plus a Golden Path developer flow. Early results show fewer mismatches (example: 192 vs 174 emails) and growing adoption, but the platform is still rolling out.
AI-written summary · about
chapters (9)
- 0:00Introduction and data cleaning costs
- 3:32From snapshots to CDC streaming
- 5:23Spaghetti architecture problems
- 10:51Strategic shift to data as product
- 15:58Event-driven architecture design
- 18:56Transaction outbox pattern
- 22:48Implementing contracts and catalog
- 26:22Cultural and technical changes
- 28:59Future of unified data platforms
related talks
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
18mData Modernization Using Confluent and Yugabyte
39mData Observability and Reliability Engineering in a Real-Time World
20mDelta Lake: Turn your Data Lake into a Lakehouse
39mDelta Lake: Diving into Data Lakes Without the Downsides
27mThe journey of Data Mesh and Compliance in a Multi-Regional Data Lake
45mThe Data Lakehouse: A tech talk for everyone who loves data
21mBuilding Data Integration Services In Real Time