Ehtsham Elahi, Senior Research Engineer, Netflix @ MLconf SEA
summary
Netflix applies Spark and GraphX to run large-scale topic-sensitive PageRank and distributed LDA, representing per-vertex activations as vectors and sampling topics via a semi-collapsed Gibbs sampler. Experiments on graphs up to billions of edges show GraphX gives sublinear time growth and overtakes a Spark+Breeze or single-machine uncollapsed implementation only after a large crossover point; GraphX suffers shuffle failures when propagating ~10k (small cluster) to ~100k (bigger cluster) vertices in parallel. Recommendation: find the crossover point, checkpoint frequently, or prefer a single large-memory machine (r3.xl) for many LDA workloads.
AI-written summary · about
chapters (8)
- 0:00ML at Netflix introduction
- 1:06ML challenges at Netflix scale
- 2:08Spark and GraphX overview
- 3:53Graph diffusion algorithms
- 7:02Distributed clustering with LDA
- 12:39GraphX performance comparison
- 16:28GraphX experiment lessons
- 17:51Closing remarks
related talks
27mData @scale Seattle - Yucheng Low
29mTed Willke, Senior Principal Engineer & GM, Datacenter Group, Intel
30mScalable Graph-Parallel Inference Algorithms and Systems
21mBuilding a Graph of all US businesses using Spark technologies
39mThorny path to the Large Scale Graph Processing
23mGraph ML – The Next Level Of Machine Learning; Learn How To Use It And When Not To
12mMake ML outcomes roar with graph neural networks on TigerGraph and AWS (PRT097)
33mMonitoring a TV streaming service with AI - from PageRank to graph convolutions