New Directions for Spark in 2015
summary
Spark claims dramatic performance and ecosystem gains in 2014 and concrete plans for 2015: contributor count grows from 150 to 500, codebase from 190k to 370k lines, and about 500 known production deployments; Databricks contributes ~75% of last year’s code. Engine work sets the 100 TB sort in 23 minutes record on 200 machines in the public cloud. Planned releases add DataFrames in Spark 1.3 and SparkR in 1.4, plus a pluggable data source API that pushes queries into MySQL, Hive and other engines.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:21What is Apache Spark?
- 1:00About Databricks
- 1:422014: an Amazing Year for Spark
- 2:23Contributors per Month to Spark
- 4:29Applications
- 4:42New Directions in 2015
- 6:10DataFrames
- 7:48External Data Sources
- 9:16To Learn More
related talks
59mSpark 2.0
19mUpdates on Spark, MLflow, and the broader ML ecosystem
11mHow Companies are Using Spark, and Where the Edge in Big Data Will Be
44mData Science at Scale with Apache Spark
54mWhat Does the Future Spark Have in Store for Us?
11mScaling R with Spark
19mWhat's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0
01h 48mBig Data made easy with a Spark (3.0)