Detecting Events on the Web with Java, Kafka and ZooKeeper
summary
Built a JVM cluster that ingests matched mentions into Apache Kafka and runs per-query in-memory spike detection for ~130,000 active queries (about 18M mentions/day for those queries), electing a worker per query via ZooKeeper leader election. Workers snapshot compacted state to HBase snapshots (≈0.3–0.4 MB/query) so failover loses only seconds, and emit detected events to a Kafka topic. Database triggers (pgq) and a Java consumer orchestrate enable/disable; adaptive backoff prevents overload, and the pipeline proves operational in production.
AI-written summary · about
chapters (10)
- 0:00Introduction and speaker
- 1:09Company and problem overview
- 9:58Why we chose Kafka
- 14:10Detecting spikes in real time
- 18:28Scaling with leader election
- 21:12ZooKeeper for coordination
- 24:21Handling long-running jobs
- 27:31Event processing pipeline
- 27:52Creating meaningful alerts
- 31:01Future plans and summary
related talks
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
19mSupercharging Kafka Applications with Real-time Contextual Insights
01h 00mPutting Apache Kafka to Use
42mBuilding a Data Pipeline with Clojure and Kafka
49mEnriching Kafka Applications With Real-time Contextual Data
38mKafka Message Broker Under High Load
18m"Kafka's Metamorphosis"
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka