Overview of the Databricks Platform
summary
Databricks presents a unified Data Lakehouse used by more than 7,000 organizations with over 3,000 employees and ~$3B funding, combining data warehouse BI and flexible data lake storage. Core pieces are Delta Lake (ACID transaction log, schema enforcement, time travel, upserts/deletes), MLflow (experiment tracking, model format, registry) and Delta Live Tables (declarative pipelines, CDC/SCD2, data quality expectations). Platform runs on Spark with the proprietary Photon engine, uses notebooks, clusters and DBUs for costed compute, and offers SQL Warehouses for BI.
AI-written summary · about
chapters (15)
- 0:00Databricks Lakehouse overview
- 1:33Lakehouse concept and open source
- 3:40Data science failures and solution
- 6:07Key platform features
- 8:14Personas and interfaces
- 10:00Spark and Jupyter foundations
- 12:30Delta Lake storage framework
- 16:48Delta Live Tables pipelines
- 18:54MLflow for ML lifecycle
- 20:16Bronze-silver-gold architecture
- 21:22Data Explorer and SQL editor
- 24:58Jupyter notebooks in Databricks
- 27:26DLT pipelines and monitoring
- 29:11AutoML and feature store
- 30:34Q&A on IDE and SQL endpoints
related talks
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
52mElevate your data and AI governance with Databricks Data Intelligence Platform
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
05mDatabricks Product Announcements
02mDatabricks at GoDataFest 2019
58mBuild reliable data lakes with Delta Lake & Databricks
26mJoel Minnick, Databricks | Databricks Data + AI Summit 2024
17mDatabricks LakeFlow: A Unified, Intelligent Solution for Data Engineering