Bringing the Jewels of the Python World to Scala with Spark
summary
Scala code calls Python NLP (spaCy/NLTK) inside Spark by flipping PySpark’s usual flow: JVM starts Python via Py4J and pipes, registers Python UDFs and ships work to Python workers. Initial pickling + sockets approach is slow; swapping to Apache Arrow reduces serialization overhead (vendor benchmark claims 242x, realistic gains ~50%). A prototype repo sparkling-ml exposes Python transformers to Scala, returns token arrays (maps not yet supported) and works at small scale but needs better failure/rescheduling for large distributed deep‑learning use.
AI-written summary · about
chapters (6)
- 0:00Start
- 1:56What Pi Spark Is
- 3:50Pi Spark Is the Python Interface To Spark
- 14:18Word Count
- 18:28Json
- 22:06Sentiment Analysis
related talks
41mKeynote: Making the Big Data ecosystem work together with Python
32mData Science in Scala with ScalaPy
33mIntegrating Scala Java with your Python code
27mPython and the JVM - A Love Story
24mPySpark - Data processing in Python on top of Apache Spark.
50mBig Data Processing and Data Science Environment in Python
43mPySpark in Practice
31mAdding Python support to Scala tooling