Machine Learning in Azure Databricks
summary
Shows end to end use of Spark ML on Azure Databricks to build a regression pipeline: read and cache a CSV from ADLS, use VectorAssembler/VectorIndexer, train a decision tree on a bike sharing (17,000 rows) dataset with a 70/30 split, save the PipelineModel and reload it, then transform unseen data to get predictions. Demonstrates that default decision tree yields repeated-value outputs and an error of RMSE ≈108 (mean 189), and explains how MLflow and hyperparameter tuning reduce error toward ~20–30.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:31Spark architecture
- 5:05How queries work in Spark
- 8:02What is Azure Databricks?
- 9:40The machine learning process
- 12:52Shallow vs. deep learning
- 17:04Types of ML models
- 21:42Components of Spark ML
- 27:13Bike sharing regression demo
- 44:56Summary and conclusion
related talks
49mAutomated Machine Learning on Azure
49mBuilding Advance Analytics pipelines with Azure Databricks
58mDeep Dive into Azure Automated Machine Learning
01h 05mAzure Machine Learning for the Developer
24mAzure ML, Azure Kubernetes Service, Model deployment
51mHow to get started with MLOps in Azure Databricks
59mPredicting the future as a service with Azure ML and R
37mForecasting using machine learning methods