What Does the Future Spark Have in Store for Us?

on YouTube ↗54m

" summary

Covers what’s new in Apache Spark 2.0: the stabilized, unified API and its multi-language support (Java, Scala, R, Python). Shows how to apply MLlib for machine learning and how to use Kafka as the standard data source for Spark DStreams to enable near-real-time processing beyond batch; intended for developers experienced with Big Data/Hadoop/Spark prior to 1.3 who want to adopt the new release.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 2:43Spark evolution changes
3. 8:44Spark 2.0 streaming
4. 15:08DataFrame vs Dataset
5. 21:06RDD optimization
6. 26:36Dataset performance
7. 32:36Memory management
8. 38:14Data formats fixes
9. 42:49Performance benchmarks
10. 48:04Tank project memory
11. 53:01Future of Spark

" related talks

• 59mSpark 2.0Matei Zaharia · Scala Days Berlin 2016
• 19mUpdates on Spark, MLflow, and the broader ML ecosystemJavier Luraschi · rstudio::conf 2019
• 25mCONTINOUS APPLICATIONS WITH SPARK 2.0Kamil Owczarek · Scalar Conference 2017
• 19mWhat's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0Reynold Xin, Tareef Kawaf · Data + AI Summit 2024
• 11mHow Companies are Using Spark, and Where the Edge in Big Data Will BeMatei Zaharia · Strata Conference 2014
• 10mNew Directions for Spark in 2015Matei Zaharia · Strata + Hadoop World San Jose 2015
• 44mData Science at Scale with Apache SparkDean Wampler · GOTO Chicago 2015
• 45mFrom Big Data to Fast Data: Apache SparkStefano Baghino · Codemotion Milan 2015