PREVIEW: Running Kafka for Maximum Pain
summary
Operational choices that accumulate technical debt and cause outages get named and quantified: LinkedIn runs about two trillion messages a day, a single cluster at 5 Gbps in / 18 Gbps out, 1,800 brokers and 2.5M partitions with clusters up to 250,000 partitions. Common failure modes include Kafka without access controls, multitenant clusters with no capacity controls, MirrorMaker inefficiencies, unreliable log compaction, shared-secret configurations and one-way upgrades, and the talk shows which of these were built, how they scale, and which still hurt.
AI-written summary · about
chapters (4)
related talks
02h 48mKafka: Map of Pitfalls for Developers and Ops
28mTackling Kafka, with a Small Team
40mKafka, DevOps...And Resilience for All
33mKafka streams at scale
40mKafka in Finance: Over 1 Billion messages a day
42mProcessing Millions of Events per Second with Apache Kafka on Kubernetes
35mKafka Internals I Wish I Knew Sooner
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)