Taming Data Skew in Production ML Pipelines
summary
Roku production Spark pipelines suddenly stretch from under an hour to ~3 hours because 95% of tasks finish quickly while one executor processes ~10x the shuffle data, causing GC and spills. Practical fixes and measured wins: repartitioning (100–200 MB partitions, 2–3× cores) yields ~30% improvement, key salting splits a hot key so an 8 GB partition becomes 8 GB -> 600 MB and training time drops ~50%, and broadcast joins cut an enrichment from 90 min -> 10 min. An AI prototype now predicts skew from historical Spark metrics and recommends partition counts, salt factors, or broadcasts.
AI-written summary · about
chapters (15)
- 0:00Intro: Handling Data Skew in Production ML Pipelines (Roku)
- 0:43Roku Scale & the Mystery of Suddenly Slower Spark Jobs
- 2:23How Skew Shows Up in Spark: Stragglers, Shuffle Spills, Idle Executors
- 4:22What Data Skew Really Is (and Why Parallelism Breaks)
- 5:39Real-World Example: Power Users, Hot Keys, and Power-Law Data
- 6:39Why It Matters: Technical Bottlenecks + Business Cost Blowups
- 8:54Where Skew Hits ML Pipelines: Recs, Classification, Computer Vision
- 10:14Root Causes of Skew #1: Natural Imbalance from Real-World Events
- 11:50Root Causes of Skew #2: Join-Key & Aggregation Skew in Feature Engineering
- 13:09Root Causes of Skew #3: Computational Skew (NLP, Embeddings, Heavy Transforms)
- 14:20Mitigation Step 1: Repartitioning—When It Works and Its Limits
- 17:04Mitigation Step 2: Key Salting to Split Hot Keys (Big Runtime Wins)
- 18:51Mitigation Step 3: Broadcast Joins to Avoid Massive Shuffles
- 21:28Wrap-Up: Choosing the Right Fix + AI to Predict Skew Before It Happens
- 23:29Closing & How to Connect
related talks
21mAvoiding Spark Pitfalls at Scale
41mSpeeding Up Data Pipeline Construction in Apache Spark
36mNo more struggles with Apache Spark (PySpark) workloads in production
40mProductionizing Big Data - Stories from the Trenches
31mSpark challenges
35mSimplification of Data Pipelines by Containerization
19mApache Beam pipelines at 100TB+ scale using Apache Spark
30mOptimize Spark Jobs Using Its UI