Data integration in the world of microservices
summary
Zalando describes a working data integration pipeline: an event-first architecture feeding a custom ziki platform that sits on Kafka and Nakadi, materializes topics to S3 via a tuang service, and exposes REST metadata for the data warehouse. It runs against ~150 microservices, replaces JDBC ETL, uses Apache Flink for stream processing and real‑time checks (Kibana → Flask), and aims for a data lake on S3; key tradeoff is losing direct DB access and trusting teams for correct events.
AI-written summary · about
chapters (7)
- 0:00Intro
- 3:30Fabians background
- 10:10Our data integration platform
- 12:00Kafka and Tukang
- 15:50Apache Flink
- 18:30Data Lake
- 20:10Open Source
related talks
40mFlink in Zalando's World of Microservices
37mData Integration in the World of Microservices
38mFrom Monolith to Microservices at Zalando
47mStreaming Data Integration for Microservices with Kafka and Kafka Connect
42mData Streaming for Microservices using Debezium
32mBusiness Intelligence in Microservice architecture #bbuzz
57mOur DataMesh Journey: data architectures for microservices
55mThe hardest part of microservices: Your Data