Taming Spark with ZIO
summary
Shows how to embed a SparkSession in ZIO’s R environment so Spark operations (save/write/sql) become ZIO Tasks, using ZIO v1 and Scalafix rules to detect uncaptured side effects. Describes refactoring an existing Spark pipeline to compile under those rules and run correctly, reporting about 5 minutes saved on a 40-minute job and improved concurrency by using ZIO’s parallel combinators.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:23Spark modules
- 4:39ZIO key concepts
- 9:47Why ZIO and Spark
- 10:29ZIO-Spark intro
- 14:07Eliminating side effects
- 14:52Practical ZIO-Spark example
- 18:11R type for environment
- 20:24Spark Session in ZIO
- 23:21Detecting side effects
- 25:14Optimizing concurrent tasks
- 27:09Async calls to services
- 28:16Conclusion
- 30:35Q&A
related talks
29mEnhancing Spark's Power with ZIO
01h 28mAsynchronous and Concurrent Programming in Scala Part 1
45mZIO: Next-Generation Effects in Scala
01h 53mAsynchronous and Concurrent Programming in Scala Part 2
01h 12mAsynchronous and Concurrent Programming in Scala Part 3
55mUnwrapping IO: is it a path that you want to follow?
59mSpark 2.0
25mCONTINOUS APPLICATIONS WITH SPARK 2.0