Holden Karau, IBM | BigDataNYC 2016
summary
Spark 2.0 migrates core work onto the Dataset API, removes legacy “junky” interfaces and enables better spilling to disk so large partitions run without OOMs; structured streaming exposes SQL-style queries on streams so aggregations and streaming ML run without manual micro‑batch plumbing. Notebooks plus Spark let data scientists operate on full-scale data (so sampling declines) and enable dataset‑recommendation workflows and rapid model retraining and deployment across production pipelines.
AI-written summary · about
