Reducing Microservice Complexity with Kafka and Reactive Streams

on YouTube ↗41m

" summary

Hootsuite migrates from PHP monolith to Scala microservices and uses Apache Kafka plus Akka Streams (reactive-kafka) to extract non‑critical work, provide at-least-once delivery and avoid blocking. Kafka scales (LinkedIn numbers: ~800B messages/day, ~118M/sec, 175TB/day; Hootsuite: hundreds of GB/day), keeps durable replicated partitions, and Akka Streams enforces full back pressure and bounded buffers. Demoed pipeline reads a committable source, uppercases records, writes to another topic, commits offsets on success — it runs and recovers correctly.

AI-written summary · about

" chapters (11)

1. 0:00Microservices transition at Hootsuite
2. 3:10Breaking monolith into microservices
3. 7:00Complexity with async messaging
4. 9:09Why Kafka for async messaging
5. 16:57Kafka message formats
6. 19:24Streaming data with Reactive Streams
7. 21:09Simplifying streams with Akka Streams
8. 25:18Akka Streams with Kafka
9. 31:19Demo: Kafka and Akka Streams
10. 35:38Summary: microservices and Kafka
11. 36:23Q&A: Kafka ordering and monitoring

" related talks

• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 44mKafka based Microservices with Akka Streams and Kafka StreamsDean Wampler · Scala Days New York 2018
• 41mApache Kafka + Apache Mesos = Highly Scalable Streaming MicroservicesKai Waehner · MesosCon Europe 2017
• 59mBuilding Data Intensive Microservices with Apache KafkaYaniv Rodenski · NDC Sydney 2017
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 30m8 Lessons Learned from using Kafka with 1000 microservicesNatan Silnitsky · Scale By the Bay 2020
• 55mBootstrapping Microservices using Akka, Kafka ans SparkAlex Silva · Scala Days Copenhagen 2017
• 40mDecomposing a live application with confidenceLuís Cobucci · Dutch PHP Conference 2018