Frameless: A More Well-Typed Interface for Spark

on YouTube ↗16m

" summary

Frameless adds a more well-typed veneer over Spark Datasets, using shapeless and cats to provide TypedDataset APIs that prove column names at compile time (selecting a missing column fails to compile). It introduces typed encoders and frameless injections to map custom types (example: sealed Genre ↔ Int) so Spark can serialize them, but API coverage is incomplete, encoded values show as ints in output, and encoders can still blow up at runtime if Spark serialization requirements aren’t met; project just graduated from incubator to full project.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 1:03What is frameless
3. 1:51RDDs
4. 2:29Example
5. 2:48DataFrames
6. 3:49DataFrame Limitations
7. 4:22Encoders
8. 5:10Columns
9. 7:32Factory methods
10. 8:32Group buys
11. 8:52Static encoders
12. 9:55frameless injections
13. 11:30Defining instances
14. 12:35Conclusion
15. 13:34Questions

" related talks

• 46mIntroduction to Apache Spark with FramelessBryan Clapper · Scale By The Bay 2018
• 06mInterviewBrian M Clapper · Scale By The Bay 2018
• 47mKeeping the fun in Apache Spark Datasets and FPHolden Karau · Scala Days Berlin 2018
• 51mThe middle way for static typing in Spark DataFramesAlfonso Roa · ScalaCon 2022
• 23mSpark Datasets: why they aren't great and what could be doneFlavio Brasil · Scale By the Bay 2017
• 47mKeeping the “fun” in Apache Spark Datasets and FPHolden Karau · Scala Days New York 2018
• 54mWhy Netflix ❤'s Scala for Machine LearningJeremy Smith, Aish · Scala Days Lausanne 2019
• 31mGoing structural with Named TuplesJamie Thompson · Scalar Conference 2025