Apache Spark Lakehouse Pushdowns with Transpilation
summary
Claim: adding type information often makes Spark 10–400% slower because typed UDFs force full-row reads, so the project transpiles Python/Scala/JVM UDFs into Spark’s Catalyst IR to enable real pushdowns. Proof‑of‑concept converts Python ASTs in the JVM, has two passing tests, and falls back to normal UDF execution on failures; it currently handles simple expressions safely but chokes on nulls, dates and JSON. Next steps target broader UDF shapes and opt‑in relaxed typing and transpile Python to Catalyst to avoid 10–400% slowdown.
AI-written summary · about
chapters (9)
- 0:00Apache Spark Lakehouse intro
- 10:19Spark DataFrames vs Datasets speed
- 15:59How pushdowns work in Spark
- 18:05Type info breaks pushdowns
- 21:56Transpilation to fix pushdowns
- 28:00Transpiling Python UDFs
- 33:13Transpilation risks and challenges
- 37:54Future transpilation plans
- 41:48Q&A
related talks
14mUDF to SQL Translation through Compositional Lazy Inductive Synthesis
29mFaster PySpark with Apache Arrow
34mSpark Performance Tuning and Troubleshooting
14mAutomated Translation of Functional Big Data Queries to SQL
40mScaling with Apache Spark (or a lesson in unintended consequences)
27mFunctional Programming on Top of SQL Engines
36mData pipelines engineering made simple with Scala
30mParticle physics, 10,000 times faster