Routing Billions of Events a Day: How we Do Routing in Schibsted
summary
Company routes ~1 billion events/day (≈780M visits), peaks above 50,000 events/sec, using collectors into Kinesis → Kafka → Spark → S3; streaming path runs a trafficker that filters/transforms with JSLT transformations and dispatches to sinks like Snowflake, Amplitude or DynamoDB. Routing config is a versioned YAML cookbook with dynamic routing reloads for immediate changes; JSON Schema enforces event types, backfilling and auditing handle failures, and metrics plus logs diagnose incidents under GDPR constraints.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:24About Schibsted
- 4:00What does a payload contain
- 4:58GDPR
- 8:10How much is 1 billion events a day
- 10:06The Poles
- 10:29Poles Echo System
- 11:49Events
- 12:31Collector Pipeline
- 13:56Streaming Pipeline
- 16:18Routing Cookbook
- 17:52Kafka Topic Architecture
- 19:06Schemas
- 20:29JSLT
- 22:47Routing Live
- 24:36Use Cases
- 26:01Schemas Versions
related talks
57mLessons in Building a System that Processes More than 70 Billion Events Daily
46mMillions of Events per Minute During a World Cup, No Big Deal!
41mScaling Telemetry Systems with Streaming
42mMake your logs work for you
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
33mHigh availability by offloading work
31mStreaming: Real-Time Architecture for Connected Products in Automotive
31mEvent Streaming on Rails