A Tale of Two Graph Engines on Spark
summary
Compares two Spark graph engines: Apache TinkerPop OLAP (Gremlin on Spark) and GraphFrames, showing concrete trade-offs. TinkerPop uses a pair RDD adjacency list so traversals start at every vertex and incur one shuffle per hop, which makes deep multi-hop traversals efficient but blows up on extremely high‑degree vertices (examples mention vertices with hundreds of millions of edges and executor OOMs). GraphFrames stores vertices/edges as DataFrames and compiles motif queries into joins, gaining Catalyst codegen and much faster counts/aggregations (demo on ~1M edges), and integrates directly with Spark SQL sources.
AI-written summary · about
chapters (7)
- 0:00Start
- 4:33Gremlin Traversal Language
- 13:28Lab Queries
- 15:37What Is the Average Star Trek Rating
- 20:37Catalyst Engine
- 30:50Performance Pro Cons
- 31:31Cons
related talks
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
29mTed Willke, Senior Principal Engineer & GM, Datacenter Group, Intel
12mTEGRA: Efficient Ad-Hoc Analytics on Evolving Graphs
55mSpark Architecture
45mSPARK-CYPHER: HOW WE WILL BE DOING GRAPH ANALYTICS FROM 2020 ONWARDS
53mLord of the Spark
42mWhy Choose One: Multi-Engine Analytics with Apache Wayang
18mEhtsham Elahi, Senior Research Engineer, Netflix @ MLconf SEA