Give meaning to 100 billion analytics events a day, analytics at Teads
summary
Claims a pipeline that handles 100 billion events per day (≈1M events/s) by writing immediately into Kafka → Dataflow → BigQuery ELT, splitting tracking into a fast collector and a Flink hydrator, then rolling up into hourly tables. BigQuery runs a roll-up query over 100M rows (25 GB) in 1m40s with ~50× compression; 60 dimensions and 150 metrics drive data marts exported into per-app Redshift clusters for sub-second web queries. Operates at scale but hits a 1 Gbps intercloud VPN bottleneck and keeps only ~10% of events for analysis.
AI-written summary · about
chapters (7)
- 0:00Start
- 1:09Why Do We Even Need Data
- 2:40100 Billion Events a Day
- 6:08Tracker Collector
- 8:47Roll-Up Job
- 10:56What Did We Learn with Bigquery
- 17:12The Core Concurrency Limit
related talks
55mCloud Analytics: from 0 to 100,000 events per second in 50'
22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
01h 01mDealing With Data Panel
38mHow to handle 100k events per second
41mEnhancing Event Analysis at Scale: Leveraging Tracking Data in Sports
11mBig Data Analytics: Immutability & Idempotence
33mManaging IoT Storage with Google's Cloud Platform