The Power of Type Classes in Big Data ETL

on YouTube ↗19m

" summary

Shows a type-driven approach to big-data ETL on Apache Spark using Scala. Demonstrates using type classes and singleton types with the Shapeless library to express data-processing logic, and builds a higher-level DSL on top of Spark so ETL code becomes easier to read.

AI-written summary · about

" chapters (7)

1. 0:00Type classes in ETL
2. 1:42Why type classes matter
3. 5:16Type-safe performance
4. 9:32Spark and ETL examples
5. 13:53Pitfalls and solutions
6. 16:00Optimizing type classes
7. 17:57Summary

" related talks

• 58mSeparating Behavior and Data with Type Classes in ScalaRaúl Raja Martínez · WeCodeFest 2018
• 20mType Classes for the Rest of UsAndrew Kuhnhausen · Scale By the Bay 2017
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 46mScala and the JVM as a Big Data PlatformDean Wampler · Scala World 2015
• 24mLet the Scala compiler work for youAdelbert Chang · Typelevel Summit NYC 2017
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 16mFrameless: A More Well-Typed Interface for SparkLong Cao · Typelevel Summit NYC 2017
• 51mThe middle way for static typing in Spark DataFramesAlfonso Roa · ScalaCon 2022