Needle in the Haystack - Monitoring health of a huge Kafka fleet with Flink
summary
Amazon MSK processes thousands of Kafka clusters’ JMX and host metrics in real time using Apache Flink running on Kinesis Data Analytics, keying by clusterID/nodeID and aggregating with tumbling one-minute windows then a five-minute sliding window to flag alarms (breach = 3 of 5 minutes). Event time preserves exact failure timestamps, missing-node detection comes from a cluster-node event source, health booleans persist to a DB, a single metric emits total unhealthy clusters, remediation workflows trigger automatically, and alarms can be suppressed or blacklisted.
AI-written summary · about
chapters (7)
- 0:00Intro
- 0:58Monitoring metrics
- 3:34Architecture
- 9:51Time characteristics
- 10:30Operational overhead
- 11:45Cluster node source
- 13:30Caveats
related talks
21mSweet Streams are Made of These: Applying Metrics-driven Development to Apache Flink
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
22mEnd-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
34mAdvanced visualization of Flink and Spark jobs
36mKafka Monitoring: What Matters!
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
33mObservability-First Kafka: Engineering Visibility at Scale