Enhancing Spark's Power with ZIO
summary
Combines Apache Spark with ZIO via an open source library zpark-io that converts Futures to ZIO, wires Spark into ZIO layers, and removes boilerplate Spark implicits; benchmarks show a toy job dropping from 40m→20m (50%) and a grown pipeline dropping from 3h→1–2h by parallelizing data fetches. Implements cancelable fibers, timeouts, retries with schedules, testable fake services via layers, and helpers for creating Datasets and broadcasts; examples and migration guide live on the project GitHub.
AI-written summary · about
chapters (33)
- 0:00Introduction
- 1:51Functional Programming
- 3:22Map
- 4:23What is Spark
- 6:20Future
- 9:38Scala
- 11:00Fuel
- 11:51ZIO Components
- 12:28ZIO Features
- 14:59Github
- 15:22The Park
- 16:58Examples
- 17:52ZIO Goal
- 18:10Unit Test
- 18:53Services
- 20:35Environment
- 21:02Runtime Environment
- 21:12Runtime Environment Example
- 21:26Shortcut
- 21:58UserId
- 22:22Scalp
- 22:41Building Spark
- 23:05Overrides
- 23:36InputOutput
- 24:07ZDS
- 24:33Data Sets
- 24:51Transform Data Sets
- 25:27Broadcast in ZIO
- 25:38Transform Futures into ZIO
- 25:57Advantages of ZIO
- 27:06Parallelization
- 27:20One Minute Left
- 27:58Outro
related talks
40mTaming Spark with ZIO
45mZIO: Next-Generation Effects in Scala
01h 28mAsynchronous and Concurrent Programming in Scala Part 1
30mNext Generation of Scala
01h 53mAsynchronous and Concurrent Programming in Scala Part 2
01h 12mAsynchronous and Concurrent Programming in Scala Part 3
32mNext-Level Diagnostics.... with ZIO
49mFIBER SUPERVISION IN ZIO