Data @scale Seattle - Yucheng Low
summary
Claims a practical stack that runs large-graph ML on one box: SFrame columnar storage holds tables on SSD with lazy, columnar access and arbitrary nested types; graphs are append-only, vertex/edge-partitioned with integer remapping and space‑filling curve placement. Compression shrinks the Common Crawl web graph from ~2 TB to ~200 GB (~10x compression), enabling PageRank on the Twitter graph (~1B edges) on a single SSD machine, and PageRank on the 128B-edge Common Crawl at ~60s/iteration on 16 cloud instances (≈2B edges/sec).
AI-written summary · about
chapters (13)
- 0:00Introduction to speaker
- 0:28From GraphLab to Dato
- 1:44The 'Dato way' of scaling
- 2:56Data representation and frames
- 4:50SFrame demonstration
- 7:49Graph data structures
- 10:03Compression techniques
- 12:32Performance benchmarks
- 13:46Computational abstractions
- 16:08Distributed graph partitioning
- 19:03Scaling results and efficiency
- 21:41Philosophy and conclusion
- 23:20Q&A session
related talks
17mCarlos Guestrin, CEO of Dato Inc. @ MLconf SEA
50mBig Data on Smallish Machines
39mThorny path to the Large Scale Graph Processing
39mThorny path to the Large Scale Graph Processing
32mLarge-Scale Graph Processing on Emerging Storage Devices
26mLarge Scale Graph Processing & Machine Learning Algorithms
09mEfficient Large Graph Processing with Chunk-Based Graph Representation Model
18mEhtsham Elahi, Senior Research Engineer, Netflix @ MLconf SEA