Lessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
summary
Databricks runs a global multi‑cloud control plane on AWS, Azure and GCP that manages millions of VMs per day, serves 5,000+ customers including >40% of the Fortune 500, and processes petabytes and hundreds of terabytes of logs daily. Engineering choices include Apache Spark, Delta Lake, MLflow, Kubernetes, Envoy/GraphQL, Terraform/Spinnaker, and optimized Scala/Bazel builds with 500x build speedups. A cloud‑agnostic service framework plus a consistency layer (an S3 commit service) and heavy automation—tens of millions of tests per day—enable rapid, declarative replication and microservice scaling.
AI-written summary · about
chapters (5)
- 0:00Introduction and overview
- 2:37Data platform architecture
- 5:03Multi-cloud challenges
- 11:03Building the platform factory
- 15:37Data-driven evolution
related talks
57mScaling Databricks to Run Data & AI Workloads on Millions of VMs
34mOverview of the Databricks Platform
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
18mDatabricks drives predictive maintenance, grid optimization with analytics
01mDatabricks Customers
40mProductionizing Big Data - Stories from the Trenches
52mElevate your data and AI governance with Databricks Data Intelligence Platform