Debugging PySpark -- Or trying to make sense of a JVM stack trace when you were minding your own bus

on YouTube ↗26m

" summary

Shows concrete patterns: search logs for the org.apache.spark.api.python Python exception, run Spark 2.2/2.3 to get better Jupyter tracebacks, and expect hundreds of duplicated JVM stack traces on large clusters (demo gives ~400 copies). Demonstrates tools: Spark History Server, SC.setLogLevel and persist to break pipelining, yarn debug_delete_debug_delay_seconds, and tuning yarn.application.memoryOverhead for JVM OOMs. Notes Arrow can be faster (vendors claim 242×), explains a broadcast+signal hack to attach a remote Python debugger, and recommends sampling locally to reproduce failures.

AI-written summary · about

" chapters (39)

1. 0:00Welcome
2. 1:28Trans Queer Canadian
3. 2:00Copresenter
4. 2:24Introduction
5. 3:14Outline
6. 3:54Key takeaways
7. 4:38What is PySpark
8. 5:06Why people come to PySpark
9. 5:28Debugging PySpark
10. 6:56Monster Identification Guide
11. 8:02Logs and Error Messages
12. 8:59Thinking about your program
13. 9:45PySpark history server
14. 10:28Change log level
15. 10:53What happens when we get an error
16. 11:10What is in the JVM stacktrace
17. 11:59Happiness is overrated
18. 12:27Error message
19. 13:02Disable error reporting
20. 13:15Upgrade Spark
21. 13:41Terminal
22. 14:10WebUI
23. 15:14Ignore Java messages
24. 15:54Console box
25. 18:50lambda expressions
26. 19:57testingbase
27. 20:22skip testing
28. 20:42Apache arrow
29. 21:11Floating point
30. 21:28Java stack trace
31. 22:09What we do
32. 22:20Out of memory
33. 23:00Summary
34. 23:27MapReduce error
35. 23:45Connecting Python debuggers
36. 24:06Running locally
37. 24:29Takeaways
38. 24:42Buy my book
39. 24:59Join me in other places

" related talks

• 40mDebugging PySpark - Pretending to make sense of JVM stack tracesHolden Karau · PyData Amsterdam 2017
• 43mPySpark in PracticeDat Tran, Ronert Obst · PyData London 2016
• 43mImproving PySpark Performance: Spark performance beyond the JVMHolden Karau · PyData Amsterdam 2016
• 41mImproving PySpark Performance Spark performance beyond the JVMHolden Karau · PyData DC 2016
• 41mKeynote: Making the Big Data ecosystem work together with PythonHolden Karau · PyData London 2018
• 24mPySpark - Data processing in Python on top of Apache Spark.Peter Hoffmann · EuroPython 2015
• 27mPython and the JVM - A Love StoryShir Havron · PyCon US 2026
• 31mBringing the Jewels of the Python World to Scala with SparkHolden Karau · Lambda World Seattle 2018