Introducing Spark Cyclone Accelerating Spark with...
summary
A Spark Cyclone plugin offloads Spark SQL to the NEC SX‑Aurora TSUBASA vector engine, claiming up to 48GB HBM2, 1.5 TB/s per card and ~6 TFLOPS (49 TFLOPS with eight cards) and integrating via Hadoop 3.3 YARN device plugin. It compiles Catalyst expressions into native C++ ahead‑of‑time, generates C++ and loads via VEO using Frovedis/Probectis, runs TPCH against a V100 (16GB) with many queries ~2× faster (one query 1s vs ~20s), while string handling and PCIe shuffles remain work items.
AI-written summary · about
chapters (11)
- 0:00Introduction to Spark Cyclone
- 1:20Project introduction and goals
- 3:49SX Aurora Tsubasa vector engine
- 5:36Vector engine architecture details
- 9:28Comparison with GPUs and alternatives
- 12:36Spark Cyclone plugin setup
- 14:25How Spark Cyclone works internally
- 17:57Code generation from Scala
- 21:06Performance comparison with V100
- 26:01Future directions and improvements
- 29:15Closing and Q&A
related talks
30mSpeed Up Your Spark Data Processing with a GPU
39mComplex Data Transformations Made Easy
05h 34mData @Scale 2024
34mHive on Spark #bbuzz
39mFlare & Lantern: Accelerators for Spark and Deep Learning
52mSolving Analytical Problems using Apache Spark
11mLightning Talk: Torch-Spyre: Compiling To a Multi-core Dataflow Accelerator...
42mApache Spark on planet scale