Apache Kafka + Apache Mesos = Highly Scalable Streaming Microservices

on YouTube ↗41m

" summary

A working demo deploys Apache Kafka on DC/OS with 3 brokers and a Kafka Streams microservice that embeds an H2O decision‑tree model to predict flight delays, creating input and output topics and using a Kafka VIP bootstrap URL for a stable endpoint. The service starts with one instance and is scaled from 1 to 5 instances via Marathon/Docker so partitions are rebalanced (round‑robin when messages lack keys); it processes each event exactly once and recovers on failure, though the Mesosphere Kafka client (0.9) requires upgrading for Kafka Streams timestamps.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and agenda
2. 1:23Microservices motivation
3. 4:27Apache Kafka ecosystem
4. 9:31Kafka Streams introduction
5. 18:14Kafka on DC/OS
6. 22:48Live demo and scaling
7. 31:30Key takeaways
8. 36:03Q&A session

" related talks

• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 33mMicroservices, Events, and Breaking the Data Monolith with KafkaBen Stopford, Cornelia Davis · Cloud Foundry Summit Europe 2019
• 41mReducing Microservice Complexity with Kafka and Reactive StreamsJim Riecken · Scala Days Berlin 2016
• 40mDeploying Kafka Streams Applications...Gwen Shapira, Matthias Sax · Scale By The Bay 2018
• 21mHow Apache Kafka® Generates Value In Retail With ConfluentAnna Semjen · Big Data LDN 2022
• 59mBuilding Data Intensive Microservices with Apache KafkaYaniv Rodenski · NDC Sydney 2017
• 31mEvent driven communication with Kafka on KubernetesRaihan Khan · KCD Dhaka 2024
• 48mBuilding a Microservice Application Using Kafka and DjangoNikolay Saskovets · PiterPy 2019