Why Netflix ❤'s Scala for Machine Learning

on YouTube ↗54m

" summary

Netflix uses Scala to encode more meaning into types, shipping a lightweight schema types layer that wraps Spark DataFrames with phantom types, runtime validation, and implicit-driven schema composition so transforms compose and the compiler rejects missing inputs. A research prototype encodes function ASTs in types as symbolic functions, computes derivatives via dependent type rules and macros, then reify to DoubleUnaryOperator at compile time; schema tooling is production-used, the symbolic work is a promising POC. Scala 3 features (opaque types, cleaner metaprogramming) are expected to remove current pain points.

AI-written summary · about

" chapters (14)

1. 0:00Start
2. 4:41Some examples
3. 5:27Schema types
4. 10:03Transform functions
5. 11:17Inferred transforms
6. 13:07Compositional flavors
7. 14:57Symbolic functions
8. 16:15Expression DSL
9. 19:07Teach the compiler algebra!
10. 20:15Teach the compiler calculus?
11. 21:00OK, a couple of macros
12. 21:53Power rule
13. 27:18Other thoughts
14. 30:41Coming soon to Scala 3

" related talks

• 39mMachine Learning with Scala on SparkJose Quesada · Scala Days Berlin 2016
• 42mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 04mInterviewShiva Chaitanya · Scale By the Bay 2017
• 35mTowards Typesafe Deep Learning in ScalaTongfei Chen · Scale By The Bay 2018
• 29mFrom encoding to machine learning a story of managing media data at NetflixArsen Kostenko · Highload fwdays'19
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 53mKeynote InterviewFaisal Siddiqi · TWIMLcon: AI Platforms 2021