Kafka Microservices with Kafka plus Akka and Kafka Streams

on YouTube ↗33m

" summary

Kafka is a distributed append-only log (not a transient message broker), gives replayable data, consumer-managed offsets, and ordering guaranteed per partition; Kafka Streams uses a Java API with KStream/KTable duality for transformations and scales by topic partitions but leaves failover and restart scripts to the operator. Akka Streams provides reactive backpressure, Alpakka connectors, actor integration and persistence so per-key work can be parallelized across threads or clusters. A simple scoring pipeline is implemented both ways (code in the repo); both work, Kafka Streams is simpler to deploy in code, Akka needs more orchestration but offers lower latency and richer integration.

AI-written summary · about

" chapters (11)

1. 0:00Start
2. 1:25Why Kafka
3. 1:35What Is Kafka
4. 6:41Streaming Engines
5. 7:45Spark Example
6. 8:54Programming Model
7. 9:58Kafka Streams
8. 12:14Processing Messages in Bulk
9. 21:56Kappa Stream Code
10. 23:28Scalability
11. 24:11Akka Streams

" related talks

• 44mKafka based Microservices with Akka Streams and Kafka StreamsDean Wampler · Scala Days New York 2018
• 50mIntroduction to Apache KafkaJames Ward · Devoxx US 2017
• 59mBuilding Data Intensive Microservices with Apache KafkaYaniv Rodenski · NDC Sydney 2017
• 55mDemystifying Stream Processing with Apache KafkaEwen Cheslack-Postava · Philly ETE 2016
• 35mBuilding a Real-time Streaming Platform Using Kafka Streams...Jay Kreps · Data By the Bay 2016
• 47mStream All the Things!Dean Wampler · Scala.IO 2017
• 41mApache Kafka simply explainedOlena Kutsenko · Berlin Buzzwords 2022
• 16mAn introduction to Apache Kafka - this is Event Streaming, not just MessagingKatherine Stanley · Voxxed Days Microservices 2019