Introduction to Apache Spark with Frameless

on YouTube ↗46m

" summary

Compares Spark DataFrame/Dataset APIs to Frameless by running live code on ~50,000 tweet samples and building typed pipelines that keep Catalyst optimizations while restoring compile‑time checks. Shows how Frameless uses typed Dataset wrappers and custom encoders (e.g., timestamp injections), produces composable frameless jobs, and avoids Dataset’s object deserialization penalty; points out tuple/projection awkwardness and some extra boilerplate but demonstrates execution plans that match DataFrame performance.

AI-written summary · about

" chapters (19)

1. 0:00Introduction to talk
2. 1:40Spark basics and setup
3. 3:45DataFrame API overview
4. 10:04Spark parallelism explained
5. 11:06DataFrame type safety issues
6. 13:40Dataset API introduction
7. 16:56Dataset performance costs
8. 24:43Frameless introduction
9. 25:47Creating typed datasets
10. 28:06Frameless transformations
11. 31:18Column operations in frameless
12. 34:37Schema and type handling
13. 36:24Aggregations in frameless
14. 39:17Adding columns in frameless
15. 40:19Frameless joins and performance
16. 41:26Nested schema handling
17. 42:30Frameless jobs and actions
18. 43:33Choosing the right API
19. 45:19Conclusion and Q&A

" related talks

• 16mFrameless: A More Well-Typed Interface for SparkLong Cao · Typelevel Summit NYC 2017
• 47mKeeping the “fun” in Apache Spark Datasets and FPHolden Karau · Scala Days New York 2018
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 45mScala and the JVM as a Big Data Platform Lessons from Apache SparkDean Wampler · Scala Days Berlin 2016
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 46mScala and the JVM as a Big Data PlatformDean Wampler · Scala World 2015
• 01mKotlin for Apache SparkPasha Finkelshtein · Data Love 2021