Flare & Lantern: Accelerators for Spark and Deep Learning
summary
Two systems that convert high‑level Scala workloads into low‑level code and claim concrete speedups: Flare compiles Spark SQL and dataflow into native kernels integrated back into Spark, showing 3–10x single‑node gains by generating vectorized, fused operators; Lantern implements tensor programs and reverse‑mode AD on the JVM, emitting efficient code for CPU and GPU and enabling fused kernels for training. Includes end‑to‑end pipeline benchmarks, kernel source generation, and comparisons that match state‑of‑the‑art DB and DL performance.
AI-written summary · about
chapters (8)
- 0:00Flare & Lantern accelerators
- 5:57Spark & DL acceleration concepts
- 12:11Operator design for records & streams
- 18:00Memory & performance optimization
- 24:06Reverse mode computation
- 30:12Virtualization & system integration
- 35:04Benchmarking & deployment challenges
- 37:37Q&A
