ML in Production on Databricks
summary
Focuses on concrete production steps: use Delta Lake for ACID files, schema enforcement and time travel; run transformations with Delta Live Tables to get orchestration, automated data quality checks and CDC; govern data and lineage with Unity Catalog; store and serve features (online stores like DynamoDB) and log models with MLflow and Model Registry so runs are reproducible. Recommends separate workspaces per environment, Databricks Runtime 13 feature tables, Canary deployments plus rollback, and automated retraining while warning that selecting production data for retrain remains hard.
AI-written summary · about
chapters (8)
- 0:00ML in Production on Databricks
- 1:07Why machine learning production is challenging
- 4:24MLOps fundamentals
- 8:22Data quality and pipelines
- 13:41Databricks ML features
- 20:44MLflow lifecycle
- 24:02AutoML and CI/CD
- 26:12Deploying models
related talks
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
44mSolving the world's toughest problems
44mML in Production
51mHow to get started with MLOps in Azure Databricks
16mKeynote: DBRX: Training Modern LLMs on Databricks
28mDataEngBytes 2023 - PER-T1-05 - Ellie Gholami
52mElevate your data and AI governance with Databricks Data Intelligence Platform
45mMLOps and ML-Flops: going from dev to prod