Scalable Data Science with Spark and R
summary
Demonstrates using SparkR DataFrame API on Apache Spark 2.0 inside a DataBricks hosted cluster: create and persist Spark data frames from R, explain lazy transformations vs actions, warn that collect pulls into R and can OOM, and run examples that actually execute — aggregations on faithful (272 rows, mean wait ≈70, most common 78 with durations 4–4.6), a glm binomial with two-class iris (two misclassifications on test), KMeans clustering, and saving models.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:57What makes Spark fast and scalable
- 10:26Setting up DataBricks
- 19:11Spark R architecture
- 26:00Exploring data with Spark R
- 39:20Interactive analysis
- 49:57Machine learning in Spark R
- 1:06:03Advanced Spark R features
- 1:16:41Q&A
related talks
17mUsing R with Databricks Connect
01h 48mBig Data made easy with a Spark (3.0)
43mIntroduction to Spark Datasets
44mProcessing large data with Apache Spark
53mTips and tricks for robust big data applications
38mPySpark in Practice
40mScaling with Apache Spark (or a lesson in unintended consequences)
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers