Advanced Kafka Features: From Dynamic Consumers to Fault Tolerance
summary
Covers advanced Apache Kafka techniques: dynamically subscribing consumers to new topics, server- and client-side message filtering, batch processing, deadlock avoidance, implementing retry topics and DLQs, optimizing producer throughput with the Kafka Producer library, applying rate limiting, and configuring high availability and cross–data-center distributed operation.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:52Notification system architecture
- 4:41Kafka challenges
- 8:16Scaling the system
- 13:18Dynamic consumer creation
- 16:13Message send semantics
- 17:55Outbox pattern for reliability
- 21:30Performance optimization
- 26:27Retry and Dead Letter pattern
- 32:33Fault tolerance with Patroni
- 35:06Summary
- 35:27Q&A
related talks
53mDemystifying Kafka
16mIntroducing Apache Kafka
05mIntroduction to Kafka Internals
44mFive Years of the Good and the Not-So-Good with Kafka
32mDetect and Automatically Fix the Most Common Apache Kafka Problems
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
41mApache Kafka simply explained
46mBrick-by-Brick: Exploring the Elements of Apache Kafka®