Building a Graph of all US businesses using Spark technologies
summary
Builds a production business index using Spark GraphX with Scala and MLlib models and claims it resolves noisy, duplicate records into real companies by clustering locations then linking them into parent businesses using connected components and aggregateMessages. Uses supervised models (logistic regression, random forest, GBD) to pick best attribute values and impute missing data and performance tweaks like checkpointing and keeping vertices in memory. Runs at scale with 250 million vertices and 1 billion edges and serves predictive marketing queries for customers.
AI-written summary · about
chapters (11)
- 0:00Introduction to presentation
- 1:18What is Radius Intelligence
- 3:22Business graph data pipeline
- 8:36GraphX overview and API
- 10:57Building location vertices
- 12:46Linking locations to businesses
- 14:52Aggregating messages in graph
- 16:38Connected components for businesses
- 17:41Performance optimizations
- 18:45Using the business graph
- 19:29Lessons learned and scaling
related talks
43mBuilding Billion Node Graphs for Machine Learning
18mEhtsham Elahi, Senior Research Engineer, Netflix @ MLconf SEA
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites
36mBig Data Modeling with Spark SQL: Make data valuable
56mGraph Analytic Systems
39mThorny path to the Large Scale Graph Processing
05mInterview
01h 36mSpark for Scala Developers