Evolution of Data Collection in the Datalake at Samokat Infrastructure
summary
Describes implementing EventStreaming to feed a DataLake at ecom.tech: why they adopted event streaming, which problems they aimed to solve, and why they rejected common CDC (Postgres WAL / Debezium). Explains the initial implementation choices, the tools selected and rationale, and how and where data was stored initially and currently, plus how they coordinate storage decisions.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:48Data delivery problem
- 6:02Direct DB connection
- 7:26Two-phase commit
- 8:07Debezium adoption
- 9:50Transactional Outbox
- 13:25Confluent Schema Registry issues
- 15:32Apicurio Registry switch
- 17:19Product data ownership
- 19:47Summary and benefits
- 22:12Alternative approaches
- 23:57Q&A
related talks
44mThe Changing Face of ETL: Event-Driven Architectures for Data Engineers
34mTales of Data Architecture Evolution
29mUnifying Real-Time and Data Lake: Yotpo's Transition from Chaos to Coherence
45mThe Data Lakehouse: A tech talk for everyone who loves data
39mHigh-volume Data Streaming and Storage
45mModern Data Architecture
28mDemocratization of Data
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics