Big Data Pipelines With Scala - Part 1 of 2

on YouTube ↗01h 03m

" summary

Claim: Spotify processes many terabytes of data per day and builds a Scala layer, Scio: a Scala API for Apache Beam, to express pipelines. Tools actually used: Apache Beam for the programming model and runs with Google Cloud Dataflow as a production runner, plus a local runner for fast testing. Surprising numbers: ~200 internal users of Scio and tiny local runs finish in seconds but production needs careful shuffle optimization. Repo and hands-on exercises include failing tests to implement.

AI-written summary · about

" chapters (22)

1. 0:00Intro
2. 1:05Who am I
3. 1:43Hiring
4. 1:58Agenda
5. 4:55Google Code
6. 5:41History of Big Data Ecosystem
7. 7:35Staking
8. 10:05Beam
9. 11:15Shear
10. 12:11How it works
11. 14:05Code
12. 14:40Data Model
13. 17:27SBT Shell
14. 18:30Run SBT Shell
15. 19:27Test
16. 24:16SBT Comments
17. 25:35Second Exercise
18. 38:18Unit 2 Functional Programming
19. 41:48Functional Programming
20. 49:33Parse
21. 55:29Shuffle
22. 56:42Join

" related talks

• 15mApache Beam and Google DataFlow IRL with Scala ScioAlick Paul · Scala.IO 2017
• 36mData pipelines engineering made simple with ScalaRaphael Claude · Scala.IO 2024 - Nantes
• 32mWhy Use Scala in a Bigdata Context?Vincent Heuschling · Scala.IO 2014
• 38mData processing with Apache BeamSourabh Bajaj · PyData Seattle 2017
• 42mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 48mData Streaming with Apache Beam / Cloud DataflowGeoffrey Garnotel · Devfest Lille 2018
• 07mTransforming Big Data into Real-Time Product InsightsNimit Bhardwaj · Conf42 Cloud Native 2025
• 42mDeciphering the way data is testedRajni Singh · Appium Conference 2021