What's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0
summary
Preview of Spark 4.0 shows Spark Connect becoming GA as the language‑agnostic gRPC + Apache Arrow binding that decouples driver and server, enabling per‑client processes, easier upgrades and debuggability. PySpark is now treated as first‑class with idiomatic APIs; Databricks reports over 5 billion PySpark queries/day on Spark 3.3+ and PySpark downloads from 200+ countries. sparklyr gains Spark Connect and UDF support so R clients run without JVM installs. A Spark 4.0 preview build is available for testing.
AI-written summary · about
chapters (7)
- 0:00Apache Spark 4.0 intro
- 1:50Python first-class in Spark
- 6:44Spark dependency challenges
- 8:08Spark Connect language APIs
- 11:18Sparklyr and R with Connect
- 15:56Spark Connect benefits
- 16:39Spark 4.0 features preview
related talks
55mUse Spark from Anywhere
19mUpdates on Spark, MLflow, and the broader ML ecosystem
54mWhat Does the Future Spark Have in Store for Us?
59mSpark 2.0
10mNew Directions for Spark in 2015
45mSPARK-CYPHER: HOW WE WILL BE DOING GRAPH ANALYTICS FROM 2020 ONWARDS
47mThe future of data: Apache Spark 3.1, Delta Lake and Koalas
46mSpark: One Tool for Everything?