A Principled Approach to Monitoring Streaming Data Infrastructure at Scale
summary
CrowdStrike ingests over a trillion events per day and models monitoring along two axes to produce four quadrants: observability, operability, availability, quality. It instruments consumer group lag via Burrow, snapshots lag into a time-series DB with Kafka Monitor to compute metrics like mean recovery time, shards a single huge Kafka cluster into multiple sharded Kafka clusters for near-infinite scaling, runs Alert Responder to auto-restart stuck consumers and escalates failures, uses K8s autoscaling on lag, and implements temporary partition-redistribution to eliminate lag hotspots — all deployed and producing measurable SLA improvements.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:16KAFKA @ CROWDSTRIKE
- 1:29FOUR QUADRANT MODEL
- 2:35OBSERVABILITY
- 2:47CONSUMER LAG MONITORING
- 3:19MONITORING LAG WITH BURROW
- 3:41STATUS & LAG FROM BURROW
- 4:09KAFKA MONITOR
- 4:44SHARDING KAFKA FOR INCREASED AVAILABILITY & RELIABILITY
- 6:10BENEFITS
- 6:48OPERABILITY
- 7:26ALERT RESPONDER
- 8:21AUTOSCALING
- 9:21QUALITY
- 9:40ADDRESSING UNEVEN PARTITION LAG IN KAFKA
- 11:06SLA BASED MONITORING
- 11:23WRAPPING UP
related talks
22mEnd-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline
41mScaling Telemetry Systems with Streaming
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
43mDistributed Monitoring: How to Understand the Chaos
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
41mMonitoring Solr at Scale
44mActionable insight with derived second order monitoring data
42mDistributed Monitoring: How to understand the Chaos