Scala SWAT Tackling 1 Billion Member Social Network
summary
A one billion–scale social graph (actually 830 million nodes and ~7 billion relationships) is imported into Neo4j (3.5 hours on a beefy VM) and user profiles are indexed in Elasticsearch; a Scala + Play API generates per-company graph subsets, tags documents and moves from single-row tagging to 6,000/sec bulk indexing, producing the first searchable batch in 14 seconds and full subsets in about seven minutes, with searches returning ~50 profiles at about 140 ms average search latency.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:58The Graph
- 1:29Graph structure
- 3:05Existing app workflow
- 6:09Weapon of choice
- 6:20First Steps: Extraction
- 6:36First Steps: Pushing forward
- 7:30Graph Persistance
- 8:10Goodbye Titan
- 9:32Deployment on AWS
- 10:27Bulk loading into ES
- 11:32Taring
- 13:21Bottleneck
- 14:51Search benchmark
- 15:59Objective achieved
- 17:13Summary & QA
related talks
44mHow We Store and Analyze a Large Social Graph
39mThorny path to the Large Scale Graph Processing
21mSerious Network Analysis Using Hadoop & Neo4j
41mOpen Source Social Applications with Apache Streams
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites
43mBuilding Billion Node Graphs for Machine Learning
39mThorny path to the Large Scale Graph Processing
56mGraph Analytic Systems