Machine Learning at the Speed of Light with Spark
summary
Demonstrates using Apache Spark to run iterative machine-learning far faster than Hadoop MapReduce by keeping data in memory and avoiding repeated disk I/O. Covers concrete workflows using RDD caching and persistence, splitting data into train/test, repeated training loops and retrieving accuracy, and uses MLlib (ALS and k‑means), Spark SQL and Streaming for ingestion. Presents measured comparisons (roughly 10× fewer machines or 3× less time in disk‑bound tests) and end‑to‑end demos of training, prediction and validation.
AI-written summary · about
chapters (8)
- 0:00Start
- 3:02Shared variables
- 6:45Spark Streaming
- 8:01Machine learning libraries
- 9:16Classification
- 11:27Collaborative filtering
- 12:34Clustering
- 13:55Spark Core performance
related talks
50mApache Spark vs Machine Learning
44mData Science at Scale with Apache Spark
39mMachine Learning with Scala on Spark
04mThe machine-learning renaissance
40mReal-time Learning with Spark
35mPySpark : Combining Machine Learning & Big Data
23mRethinking scalable machine learning with Spark ecosystem
19mReal Time Machine Learning Visualization with Spark