Fast Data pipelines with Akka Streams and Alpakka...
summary
Overview of building low‑latency, stateful Kafka pipelines using Akka Streams with the Alpakka Kafka connector, showing a concrete consume‑transform‑produce pattern with a committable source and transactional sink that batches commits on a 100 ms commit interval. Covers handling consumer group rebalances via revoke/assign hooks, Kafka transactions (idempotent producer, atomic writes, read‑committed isolation) to achieve effective exactly‑once, and notes that transactions raise end‑to‑end latency when chained despite little throughput loss; connector is production‑used and Lightbend‑supported.
AI-written summary · about
chapters (9)
- 0:00Introduction to Akka and tools
- 1:46Overview of Alpaca Kafka
- 5:08Understanding backpressure
- 9:48How backpressure works
- 14:42Implementing Akka streams
- 18:12Managing consumer groups
- 24:00Transactional systems in Kafka
- 32:10Implementing Kafka transactions
- 35:43Advanced processing and outlook
related talks
27mNavigating Streaming Infrastructure
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
50mIntroduction to Apache Kafka
45mApplications in the Emerging World of Stream Processing
01h 00mIn Depth Kafka Streams
37mConnecting Apache Flink to the World...
01h 00mPutting Apache Kafka to Use