One Million Clicks per Minute with Kafka and Clojure
summary
A Clojure consumer pipeline reads Kafka at 300K messages/sec and targets over one million clicks per minute by switching from per-message DB increments to 5,000-message batching, fixing a head-of-lazy-seq leak, and replacing clj-kafka messages with createMessageStreams plus multiple Kafka stream objects for concurrent pull-fetch. Clicks are imputed to positions via unreliable “hail mary” pings and joined against recent position windows stored in a Clojure priority-map of queues, which yields O(1) lookups and acceptable insert/decay cost — the system becomes stable, low-latency, and practical for real-time per-position click rates.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:42HOW NOT TO PROCESS
- 4:04The User Story
- 5:53Apache Kalka
- 10:34The dangers of an infinite lazy seq
- 12:20Batching and Grouping
- 14:51Not lockstepping network and CPU
- 16:23Wrapping the KalkaStream in a channel
- 17:37Multithreading
- 20:08The New User Story
- 28:10What have we learned?
- 29:00The JVM is a first-class citizen in stream processing
related talks
42mBuilding a Data Pipeline with Clojure and Kafka
46mClojure Scaling the Event Stream
35mKafka and the REPL: Stream Processing, the Functional Way
31mClojure at 4,000 msg/s-What We Learned, Loved, and Loathed
40mClojure Emerging Data Ecosystem
33mClojure Dispatches From a National Research Lab
41mActivity Stream Processing in Clojure
43mDoing Data Science with Clojure