Real time association mining in large social networks
summary
Compresses the full Twitter graph — 240 gigabytes and 700 million accounts with 20 billion edges — into a few gigabytes by computing minhash signatures, builds an LSH index with Locality Sensitive Hashing, and serves neighborhood queries in milliseconds on a laptop. Data collection is asynchronous (twisted) using many access tokens; heavy hashing is implemented in C via Cython for speed. Real‑time querying (<250 ms), half‑hour indexing, and Wikipedia‑tag recall tests show the pipeline yields usable, interpretable clusters.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:49Analyze Twitter network
- 6:21Social vs structured data
- 9:42Obtain full Twitter network
- 13:56Compress Twitter network
- 17:55Accelerate min-hash with Cython
- 19:42Query compressed graph
- 21:46Evaluate with Wikipedia tags
- 25:20Summary
related talks
42mScaling to Billions of Real-time Recommendations Daily
27mData @scale Seattle - Yucheng Low
37mLarge Scale Social Network Data Collection & Analysis...
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites
38mDynamics in Graph Analysis: Adding Time as a Structure for Visual and Statisti
40mArchitectures for Exploring Your Personal Data
28mEmbeddings
49mFrom research to production at ebay