Apache Flink Meets Apache Mesos and DC/OS

on YouTube ↗43m

" summary

Explains a concrete Mesos/DC/OS integration: Flink becomes a Mesos framework using Fenzo offer matching and a new FLIP-6 architecture that adds a long‑running dispatcher and per‑job resource bundles, enabling dynamic scale-up and scale-down of TaskManagers while jobs run. Demo wires Kafka → Flink → Kafka to flag money‑laundering when transactions sum > $10,000 using Flink event‑time windows; state checkpoints land on HDFS so workers can be swapped for rolling or blue/green upgrades. Packaging targets DC/OS for easy install.

AI-written summary · about

" chapters (26)

1. 0:00Introduction
2. 3:51Use Cases
3. 5:32MAC Stack
4. 6:21Stream Processing
5. 7:10Spark
6. 7:45Spark Streaming
7. 8:53Spark Execution Model
8. 9:37Flink Processing
9. 10:13Flink Fault Tolerance
10. 11:20Flink Delivery Guarantees
11. 12:34What does it mean for our data centers
12. 13:12The vision of Mesos
13. 13:37Challenges of Mesos
14. 14:25Cassandra
15. 15:08Flink SDK
16. 15:33Flink Scheduler
17. 15:48Flink JIRA
18. 16:15Why Flink
19. 17:20scheduler parts
20. 17:58resource manager
21. 20:42Flink library
22. 21:28New Flink architecture
23. 23:09Demo
24. 34:02Cluster Utilization
25. 35:02Conclusion
26. 35:42Questions

" related talks

• 41mIntroducing Flink on MesosEron Wright · Flink Forward Berlin 2016
• 41mRunning Apache Flink everywhereStephan Ewen · Flink Forward Berlin 2016
• 20mGetting Started with Event Stream Processing via Apache Flink on Microsoft AzureIsrael Ekpo · ApacheCon@Home 2021
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 36mApache Flink in the Cloud-native EraStephan Ewen · Flink Forward Global 2021
• 26mRunning Apache Flink on AWSRyan Nienhuis · Flink Forward Europe 2019
• 44mMulti Cloud MesosFrans van Rooyen · MesosCon Asia 2016
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016