Preview of Apache Spark 4.0 and GA of Spark Connect
summary
Spark Connect goes GA and becomes the default client protocol in Spark 4.0 while NC SQL is slated to be the standard SQL surface; the project team publishes a Spark 4.0 preview last week (not final) so the new APIs and changes can be tried now. Work spans compute and storage co-design with Delta Lake and Apache Iceberg integrations, and the release notes point to thousands of smaller features rather than one single headline change.
AI-written summary · about
related talks
55mUse Spark from Anywhere
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
01mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
47mThe future of data: Apache Spark 3.1, Delta Lake and Koalas
31mUsing Spark to accelerate your lakehouse architecture with Fabric
46mSpark: One Tool for Everything?
33mTo Spark or Not to Spark
24mA Scalable GA4GH Server Implementation