Flink in Zalando's World of Microservices

on YouTube ↗40m

" summary

Zalando builds Saiki, a Kafka-backed data integration layer on AWS that funnels thousands of event types into Flink jobs running on EC2/Docker, materializing results into Elasticsearch and CSV files fetched by Oracle via a REST exporter. Flink wins over Spark for sub-second latency and event-time windows; jobs run sliding windows from one minute to 48 hours, keep large keyed state and open thousands of windows concurrently. Primary uses are business process monitoring (order/delivery velocities, alerts) and streaming ETL to pre-aggregate and offload the monolithic data warehouse.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 1:11Zalando overview
3. 2:38Old BI architecture
4. 4:24Radical agility shift
5. 6:55Event bus with Kafka
6. 8:00Sick team and Flink setup
7. 11:57Flink in microservices
8. 13:20Stream processing choice
9. 15:10Spark vs Flink comparison
10. 18:43Flink deployment setup
11. 19:49Business process monitoring
12. 23:51Streaming ETL use case
13. 27:05Future Flink use cases
14. 31:26Conclusion and Q&A

" related talks

• 22mData integration in the world of microservicesFabian Wollert, Valentine Gogichashvili · Microservices Day London 2016
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 21mFlink's application at DidiXue Kang · Flink Forward Virtual 2020
• 47mBring Flink to Large Scale ProductionXiaowei Jiang · Flink Forward Berlin 2017