Scala and the JVM as a Big Data Platform Lessons from Apache Spark

on YouTube ↗45m

" summary

Scala-backed Apache Spark uses a Collections-like Scala API and evolves a set of pragmatic fixes for JVM limits: Tungsten builds custom data layouts, manual on- and off-heap memory management (via sun.misc.Unsafe), and aggressive code generation that moves from subexpression to whole-query bytecode, yielding up to ~10x improvements and making Python performance near Scala. Real problems documented include GC pauses (reports up to 10 hours), Java object bloat (48B vs 4B for small strings), and a REPL serialization trap hitting the 2GB array VM limit; compact row encoding and packed hash maps mitigate these.

AI-written summary · about

" chapters (11)

1. 0:00Start
2. 1:31A Distributed Computing Engine on the JVM
3. 1:54Resilient Distributed Datasets
4. 3:44Example: Inverted Index
5. 7:35SQL/ Data Frames
6. 37:17Scala REPL
7. 39:39Lightbend is investigating re-engineering the REPL
8. 40:57Spark Is Driving Scala Adoption
9. 41:29Scala should adopt some Tungsten optimizations...
10. 41:47could the JVM adopt Tungsten's object encoding?
11. 41:58The JVM needs long indexing, value types & unsigned types

" related talks

• 46mScala and the JVM as a Big Data PlatformDean Wampler · Scala World 2015
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 47mKeeping the “fun” in Apache Spark Datasets and FPHolden Karau · Scala Days New York 2018
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 28mRust and Scala, Sitting in a Tree….Evan Chan · Scale By The Bay 2019
• 48mHigh performance Scala systems without tearsZahari Dichev · Scala Days Berlin 2018
• 01h 05mScala: The Simple PartsMartin Odersky · GOTO Chicago 2014
• 40mScala and Spark in the New Deep Learning EraEmiliano Martinez Sanchez · Scala Days 2023 Madrid