Catch me if you Can Processing Terabytes in Secs
summary
Custom in‑house stream database processes crash reports from half a billion devices using only 19 cheap Azure VMs, built as Erlang front end plus an in-memory C layer with linked‑in driver and a tiny Lisp query language; it serves ~30,000 live customers, does lock‑free aggregations, lazy file loading and fast replication, and was acquired by Splunk after proving the system works in production despite VM crashes and C‑layer fragility.
AI-written summary · about
chapters (7)
related talks
39mAdvanced Stream Processing on the Edge
48mTaming & Composing High Performance Stream-Oriented Processing Systems
28mBest of Stream + Batch: When Streaming Analytics Isn’t
38mStreaming analytics and CEP ... #bbuzz
40mStream Processing and New Approaches: Edge Processing
38mDeep Dive Fluent Bit: Logging & Stream Processing
36mHow to Build a Solution Processing Terabytes of Data (...)?
30mFive Challenges in Real-time Stream Processing and Five Solutions