Joseph K. Bradley, Software Engineer, Databricks Inc. @ MLconf SEA
summary
Spark exposes a distributed DataFrame + Catalyst optimizer API (Python/Java/Scala, R coming) that runs the same code on laptop or cluster and beats old RDD code on a 10M-row aggregation benchmark; demo loads 11K 20-newsgroup documents, creates 2,300 positive / 8,900 negative labels, and pipes Tokenizer → HashingTF → LogisticRegression. A pipeline.fit → cross-validator sweep over numFeatures and regularization picks 10,000 features and reg=0.2 and raises test AUC from 0.89 to 0.93; demos run on Databricks cloud.
AI-written summary · about
chapters (9)
- 0:00Intro
- 8:50ML Pipelines
- 10:30Running Example
- 12:15Logistic Regression
- 13:37FML Pipeline
- 15:24ML Pipeline
- 19:27Summary
- 21:09Looking Ahead
- 22:16Learn More
related talks
14mTony Baer, dbinsight & Doug Henschen, Constellation Research | Databricks Data + AI Summit 2023
19mDatabricks
34mOverview of the Databricks Platform
15mSanjeev Mohan, SanjMo | Databricks Data + AI Summit 2023
01h 48mBig Data made easy with a Spark (3.0)
11mSteve Sobel, Databricks | Databricks Data + AI Summit 2023
50mApache Spark vs Machine Learning
44mData Science at Scale with Apache Spark