Recursion Schemes Without the Barbed Wire (Part 3)
summary
Shows how recursion schemes (using matryoshka) apply to a real data‑lake pipeline: schema validation and transformation into columnar formats by writing algebras/coalgebras that map a SchemaF into Spark’s schema ADT (the copied Spark DataType ADT), implemented in Scala with the byRecursive/Recursive typeclasses trick (Aux pattern). Exercises build validators and Parquet/Avro mappers that mostly work in demos, with a few compiler/implicit and stack‑safety caveats noted.
AI-written summary · about
chapters (10)
- 0:00Recursion schemes demo intro
- 2:53Quicksort with Matryoshka
- 7:53Recursive vs corecursive types
- 14:50Fixed point types with Recursive
- 18:01Schema definition modeling
- 21:39Exercise overview
- 24:13Schema evolution pipelines
- 26:59Show instances with Show$Delay
- 28:34Struct fields formatting
- 31:46Q&A
related talks
36mData pipelines engineering made simple with Scala
29mSchemas: The Key To Data Happiness
42mScala: The Unpredicted Lingua Franca for Data Science
43mIntroduction to Spark Datasets
32mWhy Use Scala in a Bigdata Context?
30mContracts, Schema Evolution in Data Pipelines
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake
29mData Acquisition: Overcoming Challenges