Stream processing with Kafka in Big Data environments
summary
Covers practical issues when running Apache Kafka consumers and producers in production for Big Data systems, naming Java as the implementation language. Describes common Kafka Consumer errors, their causes, metrics to detect problems, and concrete fixes including Java-side optimizations and Kafka consumer configuration tweaks to improve performance in high-load environments.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:04Speaker and topic
- 3:12Stream processing challenges
- 10:16Kafka performance issues
- 13:27Classic approach drawbacks
- 14:31Parallel consumer concept
- 21:35Parallel consumer setup
- 25:26Parallel consumer drawbacks
- 29:35Resource utilization solutions
- 37:17Consumer and producer mistakes
- 44:39JVM and parameter optimization
- 49:17Summary and recommendations
related talks
50mApplication development and data in the emerging world ...
01h 29mEvent stream processing using Kafka Streams
27mStream processing with Kafka streams
45mApplications in the Emerging World of Stream Processing
41mKafka as a Platform: the Ecosystem from the Ground Up
16mAn introduction to Apache Kafka - this is Event Streaming, not just Messaging
50mSpring Kafka beyond the basics
57mWhen Everything Went Kafka