Fault Tree Analysis Applied to Apache Kafka
summary
Provides numerical fault-tree answers for Kafka choices: uses AFR disk rates (rotational 4%, SSD 0.8%), network 2%, hardware/OS 1%, and remediation windows to compute availability and durability. Finds two brokers + three Zookeepers gives ~3.5 nines; moving to replication factor 4 or SSD/EBS reaches 4 nines ≈ 43 minutes/year; small brokers and horizontal scale improve durability vs huge 48 TB brokers. Includes tooling and diagrams in github.com/andreyfalko/FTA-Kafka and rakiaoff/scram and reports that the method produced reliable clusters quickly.
AI-written summary · about
chapters (29)
- 0:00Intro
- 4:11Outline
- 5:04Kafka
- 6:01Observability
- 7:15Event Streaming
- 8:10Change Data Capture
- 9:14Why Kafka
- 10:22Reliability Thresholds
- 10:58Availability
- 12:17Fault Tree Analysis Basics
- 13:41Event Symbols
- 14:54Gate Symbols
- 15:40Raid 0 Array
- 17:15Probability Theory
- 20:18Success Probability
- 20:37Failure Probability Formula
- 21:21Drying Fall Trees
- 23:18Zookeeper
- 25:52Other scenarios
- 27:21General guidance
- 28:53Data Durability
- 30:23Replication Rate
- 32:40Durability
- 33:03Moral
- 33:24Wait and See
- 34:08Resources
- 34:59Things to take away
- 35:35Looking in for the future
- 37:07Outro
related talks
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
30mWhy Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)
53mDemystifying Kafka
38mRe-architecting Apache Kafka for the Cloud
46mKafka 4, Fantastic?
45mAdvanced Kafka Features: From Dynamic Consumers to Fault Tolerance
38mKafka Message Broker Under High Load
46mFault tolerance in Distributed Systems: A Case Study in Apache Spark