Data pipelines engineering made simple with Scala
summary
Claims a production Scala platform that uses a SQL-like language to statically infer dependency graphs and output schemas, reducing backfills and schema surprises; it transpiles queries to Spark SQL and runs on Criteo’s Hadoop fleet (3,000 nodes, ~200,000 tables), scheduling ~1 Spark job/sec across ~5,000 datasets. Implementation details: lexer/parser with fastparse v1 and parser combinators, formatting with Pages, validation with cats Validated/NonEmptyList, concurrency with FS2 and cats-effect. Project exists in production for years and resolves missing partitions by comparing theoretical vs actual listings.
AI-written summary · about
chapters (9)
- 0:00Data pipeline challenges
- 5:23Traditional scheduling problems
- 9:21SQL-based pipeline design
- 14:07Parsing SQL with combinators
- 17:26SQL query semantics
- 21:44Dependency resolution
- 24:59Scheduling with FS2
- 28:39Key libraries & takeaways
- 30:40Q&A
related talks
18mScalable, Code-Free ETL
42mScala: The Unpredicted Lingua Franca for Data Science
01h 03mBig Data Pipelines With Scala - Part 1 of 2
32mWhy Use Scala in a Bigdata Context?
57mBig Data Pipelines With Scala - Part 2 of 2
42mDeciphering the way data is tested
28mAn Architecture to Tweet Them All
50mChallenges that everyone struggles while productionizing Apache Spark workloads