Sparklens: understanding the scalability limits of Spark applications
summary
Sparklens (OSS github.com/qubole/sparklens) uses Spark event listeners to compute a job’s critical path and ideal time and to simulate scaling effects; it flags driver-side work, lack of tasks and skew. On a POC a real job drops 158→26→10 minutes after fixing serial writes and repartitioning; critical path numbers move from 127 to ~25 to ~7 minutes while the ideal time reaches ~5–7 minutes. The tool reports executor waste and predicts whether adding executors helps.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:03Performance Tuning Principles
- 3:52Spark Application
- 6:26Doing Nothing
- 7:56Driver Side Computation
- 11:33Not enough tasks
- 12:12How to control number of tasks?
- 13:54Non-uniform tasks: Skew
- 14:38What is skew?
- 15:46Critical Path
- 18:03What have we learned so far?
- 18:56What is Sparklens?
- 21:00Sparklens: First Pass
- 22:18Observations & Actions
- 23:17Sparklens: Second Pass
- 24:50Sparklens: Simulation
- 26:27Sparklens: Per Stage Metrics
- 27:34Key Per-Stage Metrics
- 30:16Limitations
- 33:06Summary
- 33:26Using Sparklens in your application
related talks
34mAdvanced visualization of Flink and Spark jobs
29mScaling Apache Spark on Kube to Apple Scale
40mProblem Solving Recipes Learned from Supporting Spark
38mPySpark in Practice
44mProcessing large data with Apache Spark
54mRunning Remote Shuffle Service to Solve a Well-Known Challenge for Apa...
19mReal Time Machine Learning Visualization with Spark
36mScaling data workloads using the best of both worlds: pandas and Spark