Discovering Types (from Strings) with Cats and Shapeless

on YouTube ↗18m

" summary

Builds a schema discovery pipeline that takes one CSV column of strings and guesses concrete types using Cats and Shapeless rather than macros It implements parsers that return Option[T] for String Double Date etc and preserves those types by storing parsers in an HList of Parsers and invoking a Poly function to run them The pipeline also aggregates categorical counts by folding single entry maps with a Map monoid aggregation and demonstrates the approach works end to end in the provided REPL code

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 0:18The problem
3. 1:00Data scientists love strings
4. 2:19Heuristics
5. 3:00Demo
6. 3:15Explanation
7. 3:28Parser
8. 5:46Shapeless
9. 7:06Pars
10. 7:52Strings
11. 12:23SemiGroups
12. 13:09Combine
13. 14:11MonoID
14. 14:46Complete Code
15. 15:04Implicit Parameters
16. 15:44Questions

" related talks

• 35mScalable data pipelines with shapeless and catsMarcus Henry, Jr. · Typelevel Summit NYC 2017
• 33mTYPE CLASSES FROM THE GROUND UPNicolas Rinaudo · Scalar 2020
• 57mKittens: Shapeless Typeclass Derivation for CatsMiles Sabin · Scala World 2015
• 20mType Classes for the Rest of UsAndrew Kuhnhausen · Scale By the Bay 2017
• 31mCool toolz in the Scalaz and Cats toolboxesJan Pustelnik · Scalar 2016
• 49mRoll Your Own ShapelessDaniel Spiewak · Scala Days Berlin 2016
• 36mKittens - datatype-generic functional programming with ScalaKailuo Wang · Strange Loop 2016
• 24mLet the Scala compiler work for youAdelbert Chang · Typelevel Summit NYC 2017