How to handle 100k events per second
summary
Bol.com measures at the source (server or owning service) to capture full attributes per event so joins and cache races can't change cause/effect, handling about 100k events/sec and over 1 billion measurements/day. The pipeline uses Apache Kafka + Apache Flink to preserve per-session ordering and provide low‑latency, exactly‑once, keyed stateful processing. Messages use Avro with a schema registry fingerprint for bi‑directional evolution. Session routing pins a visitor to one instance, simple pushdown automata handle causal patterns, and the system runs in production for personalization, dashboards and fraud prevention.
AI-written summary · about
chapters (6)
- 0:00Introduction to measuring 2.0
- 7:19Why JavaScript measurements fail
- 14:03End-to-end ordering guarantees
- 21:32Schema evolution with Avro
- 28:08Real-world use case demo
- 31:23Recommended resources and Q&A
related talks
57mLessons in Building a System that Processes More than 70 Billion Events Daily
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
47mSpeeding Event Processing to 1.6M/sec: Badoo’s Experience
01h 01mDealing With Data Panel
24mGive meaning to 100 billion analytics events a day, analytics at Teads
50mHow We Store 60,000 Events per Second
34mChasing the Stream Processing Utopia
48mHyper Speed! When Big Data Blooms