YTsaurus SPYT: how we got rid of an Apache Spark fork
summary
Describes why the team originally forked and patched Apache Spark for the YTsaurus SPYT project, what operational and maintenance inconveniences those changes caused, and how they removed the fork by supporting SPYT on arbitrary upstream Apache Spark versions to solve the main challenges after open-sourcing the project.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:59Reasons for Spark fork
- 9:16Fork problems and new needs
- 12:52Monkey patching method
- 19:54Dropping the fork for original Spark
- 25:35Ensuring binary compatibility
- 31:32Future plans and conclusions
related talks
40mYTsaurus SPYT: Helping the Apache Spark Scheduler Be Even More Efficient
11mThe Evolution of Apache Spark™
19mWhat's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0
46mSpark: One Tool for Everything?
36mNo more struggles with Apache Spark (PySpark) workloads in production
03mFred Reiss Interview
31mKotlin for Apache Spark: Why the world needs one more API for Spark
42mWhy Choose One: Multi-Engine Analytics with Apache Wayang