Flare: Scale Up Spark with Native Compilation and Set Your Data on Fire
summary
Flare compiles Catalyst query plans to native C and runs them with NUMA-aware scheduling and data layouts, turning Spark SQL jobs that took ~110 seconds on a 2 GB CSV into native runs of ~3–6 seconds after a ~200 ms compile step; hand‑written C showed similar 3 s runtimes, Scala code about 18 s. On TPC‑H SF10 Flare delivers order‑of‑magnitude speedups versus Spark 2.1, matches other query compilers, and scales to many cores with architecture‑aware partitioning.
AI-written summary · about
chapters (12)
- 0:00Start
- 16:52Spark Architecture
- 18:04Flare: a New Back-end for Spark
- 25:27How do Query Engines Work?
- 27:51Practical Realization
- 29:10Staged Query Engine with LMS
- 31:27Data Structure Specialization
- 31:40Single-Core Running Time
- 32:36Apache Parquet Format
- 33:15Parallelism
- 35:29NUMA Optimization
- 37:34Flare Level 3 Evaluation
related talks
36mFlare: Optimizing Apache Spark for Scale-Up Architectures and Medium-Size Data
28mAn Apache Spark query's journey through the stack
30mParticle physics, 10,000 times faster
01h 32mBlazing Performance with Flame Graphs
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
36mScale your data, not your process: Welcome to the Blaze ecosystem
21mLightning fast cluster computing with Cassandra and Spark