Detect and Automatically Fix the Most Common Apache Kafka Problems
summary
A lightweight network proxy for Apache Kafka inspects and rewrites the Kafka protocol to enforce rules like topic name patterns, max partitions, replication factor, and retention. It can force producer/consumer settings (linger.ms, compression, acks), block bad topic creations, simulate chaos (duplicate records, added latency) for testing, runs as Docker or SaaS, supports any client language, and has been used to isolate 4,000 tenants on one cluster with only a few milliseconds extra latency.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:22Kafka common problems
- 3:25Critical Kafka config
- 5:56App error handling
- 7:22Antipatterns and best practices
- 9:51Human limits
- 12:15Kafka proxy solution
- 17:13Policy implementation
- 20:42Message interception and validation
- 23:12Kafka proxy benefits
- 24:33Chaos testing and transformation
- 26:39Demo and implementation
- 27:43Call for contributions and Q&A
related talks
16mIntroducing Apache Kafka
45mAdvanced Kafka Features: From Dynamic Consumers to Fault Tolerance
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
50mIntroduction to Apache Kafka
41mApache Kafka simply explained
02h 05mA Deep Dive into Apache Kafka This is Event Streaming
53mDemystifying Kafka
05mIntroduction to Kafka Internals