YTsaurus SPYT: Helping the Apache Spark Scheduler Be Even More Efficient
summary
YTsaurus SPYT describes using the YTsaurus storage and its SPYT integration to reduce or avoid expensive Apache Spark shuffle stages during GROUP BY and JOIN operations. The talk explains when shuffles are unnecessary and how YTsaurus/SPYT techniques and planner adjustments can make Spark query execution more efficient.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:10Apache Spark
- 2:16YTsaurus and SPYT overview
- 3:22Spark scheduler
- 5:29Shuffle problem in distributed computing
- 7:42Shuffle optimization for sorted data
- 9:53Spark sorting
- 12:40Custom logic in Spark plan
- 15:57Scheduler deadlock problem
- 17:49Aggregation key duplication
- 19:14Spark Join
- 21:49Join optimization for sorted data
- 24:39Join sort mismatch cases
- 26:48Testing optimizations in practice
- 27:57Optimization results summary
- 28:42Q&A
- 30:26Comparison with Spark native methods
- 31:31Uneven partition problem
- 33:17Adaptive scheduling interaction
- 34:01Testing on SSD and integration plans
- 35:48Sorted tables performance impact
- 36:54Anchor data model and YTsaurus
- 39:03Conclusion
related talks
39mYTsaurus SPYT: how we got rid of an Apache Spark fork
36mNo more struggles with Apache Spark (PySpark) workloads in production
04mDBTB INT Sandy Ryza r
24mImproving interactive querying experience on Spark SQL
30mOptimize Spark Jobs Using Its UI
30mImproving PySpark Performance: Spark performance beyond the JVM
34mHow to drive a million kilometers in YTsaurus
15mOptimizing Query Performance in Big Data Systems