Data platforms synchronisation journey at Carrefour
summary
Carrefour builds an open‑source service Event Sync (about 500 lines of Go) to aggregate readiness signals from autonomous country platforms and trigger central ingestion into BigQuery, because no cloud product offers a flexible fan‑in. It runs on Cloud Run + Pub/Sub + Firestore, supports observation windows (10–24h), minimum occurrence rules and “keep last” semantics, demos successfully with France, and is partially deployed but not yet global.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:49Cloud Run and serverless processing
- 3:12Decentralized data landscape
- 7:05Synchronizing with Cloud Scheduler
- 10:40Parallel processing with DBT
- 12:43Event-driven with Pub/Sub
- 14:47Global sync time zones
- 17:34Fan-in pattern benefits
- 19:21Workflows proposed solution
- 22:09Introducing Event Sync
- 24:21Event Sync architecture
- 28:56Live demo of Event Sync
- 33:12Adoption and future plans
- 34:54Q&A
related talks
41mThe Different Synchronization Modes of Data Platforms at Carrefour
33mBuilding an Elastic, Resilient Data Layer to Power New Retail Experiences
46mKafka Streams at the Big Data Crossroads at Lightning Speed
33mManaging IoT Storage with Google's Cloud Platform
37mBuilding Scalable Apps with Cloud Firestore
37mBuilding a Data Platform from scratch
38mCloud Microservices to Serve the Next Billion
42mDatabases unveiled; choose wisely