"Delayed Data" — Our Mechanism for Ensuring Consistency

on YouTube ↗47m

" summary

Describes a migration from a homemade data bus to Apache Kafka and Go for inter-service data exchange to solve message loss and single-threaded import bottlenecks. Explains a mechanism called “deferred data” used to ensure consistency between services during that transition and how Kafka+Golang are applied to implement it.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 8:22Data integration problems
3. 15:34Delayed data solution
4. 22:30Data import optimization
5. 27:07Summary
6. 29:41Q&A

" related talks

• 59mBuilding Data Intensive Microservices with Apache KafkaYaniv Rodenski · NDC Sydney 2017
• 48mIntegrating microservices by accepting the data dichotomyDamien Gasparina · Voxxed Days Microservices 2018
• 49mHow to Finish a Migration Project? What If We Used Kafka?Arnaud Esteve · DevFest Lille 2023
• 30mChange Data Streaming Patterns in Distributed SystemsGunnar Morling, Hans-Peter Grahsl · Berlin Buzzwords 2021
• 21mHow Apache Kafka® Generates Value In Retail With ConfluentAnna Semjen · Big Data LDN 2022
• 55mThe data dichotomy: Rethinking data and services with streamsBen Stopford · NDC London 2018
• 57mEvent Driven CollaborationIan Cooper · NDC Oslo 2019
• 23mDerisking Migrations of API BackendsSushant Bhadkamkar · GopherCon Europe 2021