How We Replaced Hundreds of Joins with One Real‑Time Processing Pipeline at 1M RPS
summary
Describes replacing hundreds of JOINs with a single real-time processing pipeline to handle 1 million RPS for ad events. It explains building a system that maintains precise, up-to-date information about each advertising event and its multi-day history across all parts of the advertising stack using high-throughput real-time processing technologies.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:51Data funnel transfer issues
- 3:37JOIN drawbacks
- 6:47Stateful streaming and KV storage
- 9:57Storage optimization and load balancing
- 13:30Reliability issues and backups
- 17:43SSM-trees for resource savings
- 21:57Implementation results and system benefits
- 25:58Q&A
related talks
41mLessons Learned Implementing a Resilient, Distributed, Real-Time Architecture
34mHow We Handle Over 1 Trillion HTTP Requests ...
36mIn-memory data pipeline and warehouse at scale
44mHow to Handle Over 1 Trillion HTTP Requests/day: Behind the Scenes of the Ad Tech Industry
24mUnder Pressure: Applying ML in Real Time
32mBig Data Processing : Where do we go next ?
34mA journey of scaling high-volume low-latency stateful processing
23mReal time Bidding at Scale: A Deep Dive into Sub 40ms Strategies