Scaling Databricks to Run Data & AI Workloads on Millions of VMs
summary
Databricks runs millions of VMs across ~7,000 customers processing exabytes per day and builds a Scala control plane on Kubernetes with Envoy and Prometheus; it ships autoscaling and auto-shutdown in Apache Spark and a transactional cloud storage layer called Delta Lake. Real failure example: a customer’s Lambda uploaded a million files, the job service exceeded a cloud VPC limit of ~1,000 TCP connections, causing GC and health‑check timeouts that aborted other jobs. Testing strategy: stress each dimension (users, rows, connections) until failure, record behavior, then fix.
AI-written summary · about
chapters (28)
- 0:00Intro
- 1:49Outline
- 2:36About Databricks
- 4:12Our Product
- 9:08Example Use Cases
- 11:09Usage Growth: VMs/day
- 12:08What's Challenging About This?
- 15:45Benefits of Cloud Software for Users
- 17:05Challenges in Building Cloud Software
- 19:34Challenges of Large Scale
- 20:26Four Lessons
- 22:06Causes of Significant Outages
- 26:24Example Outage: Aborted Jobs
- 33:07Surprisingly Rare Issues
- 37:10Testing for Scalability & Stability
- 39:04Example Scalability Problems
- 41:31Databricks Stress Test Infrastructure
- 43:15Example Output
- 44:23Developing Control Planes
- 44:43Service Framework
- 45:28Best Practices
- 47:54Example: Cluster Manager
- 49:58Delta Lake Motivation
- 51:27Example: Atomic Parallel Writes
- 52:58Delta Lake Design
- 54:35Major Benefits of Delta Lake
- 55:15Launched Today: SQL Analytics Product
- 55:46Conclusion
related talks
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
40mProductionizing Big Data - Stories from the Trenches
18mDatabricks drives predictive maintenance, grid optimization with analytics
02mDatabricks at GoDataFest 2019
34mOverview of the Databricks Platform
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
01mDatabricks Customers
43mAnalyzing Big Data in Azure cloud using Spark and Databricks