Scio data processing nirvana at Spotify

on YouTube ↗24m

" summary

Scio is Spotify’s Scala API for Dataflow and after two years in production it replaces Crunch for most pipelines, handling >200 TB/day with ~3,000 jobs. A rewrite of user-key joins uses a predicted-active-user off-heap hash map (Sparky) plus selective shuffle joins, cutting cost by ~50%. Replacing Cryo with compile-time derivation via Magnolia and Beam coders yields about 25% faster jobs (60→45 minutes) and smaller serialized output.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to Scio
2. 1:39Spotify data operations
3. 3:24Scio framework overview
4. 7:48Pipeline optimization case
5. 12:03Improvements in Scio 0.7
6. 17:39Benchmarks and results
7. 23:31Conclusion and resources

" related talks

• 38mScaling ScalaJulien Tournay, Filipe Regadas · Scala Love in the City 2021
• 36mData processing at Spotify - Why portability mattersJulien Tournay, Filipe Regadas · Flink Forward Global Virtual 2020
• 04mNeville Li InterviewAlexy Khrabrov, Neville Li · Scale By the Bay 2017
• 32mData-Shuffler: Bringing Order to The Galaxy (of Data Process)Luke Hospadaruk · Clojure/conj 2018
• 39mComplex Data Transformations Made EasyNikita Ogievetsky · Data Con LA 2020
• 36mIn-memory data pipeline and warehouse at scaleEma Iancuta, Radu Chilom · Berlin Buzzwords 2015
• 54mData engineering & DataOps: Ask Me AnythingLars Albertsson, Ellen Friedman · Berlin Buzzwords 2021
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016