Clojure Scaling the Event Stream
summary
High‑availability real‑time pipeline that processes billions of events per day with 10–100 TB at rest, built with Kafka for ingest, Storm for compute, Cassandra for storage and Clojure as the orchestration and transformation layer; an initial Java attempt stalls but the Clojure system works, scales an order of magnitude, and sustains observed throughputs around 5,000–50,000 events while preserving idempotent writes and enabling data‑first evolution using pipeline‑async and instaparse and CCM.clj.
AI-written summary · about
chapters (11)
- 0:00Scaling the Event Stream Derek Troy-West
- 4:311985: Sinclair Basic
- 10:56Requirements
- 14:09Observations
- 17:57Architecture
- 26:48The Killer App
- 28:47Highlights: Core Clojure
- 30:42Highlights: Core Async
- 33:36Highlights: Instaparse
- 44:43Static Types Saving the Day
- 45:13Team Size
related talks
42mBuilding a Data Pipeline with Clojure and Kafka
41mActivity Stream Processing in Clojure
31mClojure at 4,000 msg/s-What We Learned, Loved, and Loathed
31mOne Million Clicks per Minute with Kafka and Clojure
26mMetrics Stream Processing Using Riemann
43mClojure in live sports television
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
26mBuilding Reliable, Scalable Services with Clojure and Core.async