Scaling to Billions of Real-time Recommendations Daily
summary
Twitter runs an in-house multi-tenant graph partition system that holds both forward and transpose follow graphs on nodes, powering a low-latency fan-out pipeline to turn firehose events into push-worthy candidates. Triangle-finding runs as MapReduce for mutual-follow candidates, edge-heavy signals (edge-based features) prove most predictive but costly to compute, and experiments (quarter-long holdbacks) show large engagement gains from receiver-side changes. The system balances latency vs relevance, shards by target user to handle celebrity spikes, and serves real-time notifications at production scale.
AI-written summary · about
chapters (20)
- 0:00Intro
- 5:01Anatomy of a recommender
- 6:26Experimentation
- 7:40Leading Indicators
- 10:36What to improve?
- 12:01What are challenges?
- 13:45Account recommendations
- 17:17Mutual Follows
- 19:03Triangular Loops: take-home puzzle!
- 20:34Experiment!
- 21:28Network effect
- 23:58Real-time notifications
- 28:34Pull based solution
- 30:37How to do real-time recs?
- 31:15Graph Service: Cross partitioning
- 33:34Graph Service: Multi-tenancy
- 34:38Graph Service: Multi-tenant
- 36:18Example: Social Context queries
- 37:04Example: Live notifications
- 37:46Example: MagicRecs fanout
related talks
52m100% Big Data. 0% Hadoop. 0% Java.
28mThe Evolution of Twitter's Edge
29mReliable, High Scale Tensorflow Inference....
31mReal time association mining in large social networks
22mThe Open Source Technology Behind a Tweet
30mScalability Challenges in Big Data Science
25mReal-Time Fraud Detection at Scale
32mFeed generation at ShareChat