10 Billion a Day,100 Milliseconds Per: Monitoring Real Time Bidding

on YouTube ↗48m

" summary

AdRoll bids for 42 billion requests/day with a firm 100 ms deadline, using Erlang VM instrumentation and exometer piped to statsd + DataDog and long term HBase. Real incidents: periodic timeout spikes trace to gzip background jobs preempting scheduler threads and are fixed by CPU shielding; another outage is caused when 1.6 billion cookie matches exceed an exchange’s undocumented 1.5 billion limit and trigger throttling. Emphasis is on precise, aggregated metrics, per-node alerts and automated feedback into autoscaling and throttling.

AI-written summary · about

" chapters (51)

1. 0:00Introduction
2. 1:23Exchanges
3. 2:14AdRoll
4. 2:38First Party Data
5. 3:33RealTime Bidding
6. 3:54Low Latency
7. 5:40Complex Systems
8. 7:04Rare Events
9. 7:27Bad Things Happen
10. 7:40Not So Bad Things Happen
11. 8:02The American Dust Bowl
12. 9:07The Problem
13. 9:29Observing Systems
14. 9:50Instrumenting
15. 10:56Monitoring
16. 11:22Critical Engineering Culture
17. 11:58Positive Feedback Loop
18. 12:11NASA X15
19. 13:07XOMeter
20. 15:06XOM Terms
21. 16:05XOM Functions
22. 18:17Reporters
23. 20:09Subscriptions
24. 21:03Histogram
25. 21:52XO Meter
26. 23:14Abnormal System Behavior
27. 24:18Case Study Timeout Spike
28. 25:00Timeout Spike Graph
29. 25:49System Load
30. 26:07Network Traffic
31. 26:43CPU Shield
32. 27:48Exchange Throttling
33. 28:21Bad Bid Request
34. 28:55Bid Request Differential
35. 29:26Data Analysis
36. 31:17Lessons Learned
37. 32:04Too Much Information
38. 33:07Indirect Knowledge
39. 33:22Apollo 13 Disaster
40. 34:15Instrumentation Monitoring
41. 35:08Possible Solutions
42. 36:33Possible Strategies
43. 37:22Culture
44. 38:08Overconfidence
45. 38:42Working to the Same End
46. 39:18Know Basic Mathematics
47. 39:47Tight Coupling
48. 41:00Measure
49. 41:19Empirical Problem
50. 41:47Building Complex Critical Systems
51. 42:27Managing Massive Data

" related talks

• 34mHow We Handle Over 1 Trillion HTTP Requests ...Benjamin Degerbaix, Diana Cibu · Voxxed Days Bucharest 2025
• 01h 02mAchieving High Load in Advertising TechnologyPeter Milne · Build Stuff 2017
• 44mHow to Handle Over 1 Trillion HTTP Requests/day: Behind the Scenes of the Ad Tech IndustryDiana Cibu · Codemotion Madrid 2026
• 11mData science: Next-gen performance analyticsKen Gardner · Velocity 2016 New York
• 23mReal time Bidding at Scale: A Deep Dive into Sub 40ms StrategiesDeepanshi Dhamija · Great International Developer Summit 2024
• 44mAdvertising at the Speed of Light: DMP PlatformSergey Zhemzhitskiy · HighLoad++ 2014
• 27mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million VisitorsWolfram Wingerath, Felix Gessert · Flink Forward Global 2021
• 45mReal Time Analytics with MongoDBIngo Dyck, André Peglow · Confitura 2013