RTB and Big Data - where Erlang and Hadoop meet
summary
A real RTB pipeline that actually hits scale: an Erlang RTBSelector fleet spawns per-auction processes, uses ETS for campaign state, ZeroMQ + Thrift to route requests and pools long‑lived HTTP sockets to DSPs, while a Hadoop stack (Pig/MapReduce, Hive) and Vertica-backed reporting ingest the wins. Collectors are custom Java ring‑buffer services writing Avro‑style binary files to S3. Deployed in production for years and handling up to 60 billion HTTP requests per day, the system reports horizontal scaling and workable throughput.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 0:45What is online advertising
- 1:51What is realtime bidding
- 2:36How realtime bidding works
- 4:15Realtime bidding exchange
- 4:51Realtime bidding exchanges
- 5:21Auction process
- 6:35ETS tables
- 6:56EKS tables
- 8:43DSPs
- 9:28Sockets
- 11:27Auction
- 12:27Scanning
- 13:03Rolling the system
- 13:16Maximum production
- 15:14Single request
- 16:27Onetime
- 24:37MapReduce
- 25:21Contact problem
- 25:37Mapping
- 25:54Shuffle
- 26:24Business logic
- 26:57Parallel systems
- 30:19MapReduce jobs
- 32:55Whats next
- 33:57Java C
related talks
42mAutomation Testing in a Real Big Data Project
36mFrom Machine Learning Startup to Big Data Company
39mUsing Erlang, Riak and the ORSWOT CRDT at bet365 (...)
32mBuilding Real-time Systems with Open Source Technologies
46mEvents Analysis with Riak, Perl and Erlang at Booking.com
48mHyper Speed! When Big Data Blooms
48m10 Billion a Day,100 Milliseconds Per: Monitoring Real Time Bidding
01h 02mAchieving High Load in Advertising Technology