Migrating From Single-Node Kubernetes Control Plane To HA In Product...
summary
Databricks migrates a fleet of 60 regions from a single-VM control plane to a three‑replica HA design, using etcd snapshot and disk clone to preserve cluster state and replacing VM IPs with public/internal load balancer IPs so no client reconfiguration is required. Migration runs per cluster in an automated pipeline (10–30 minutes), upgrades tolerate one node/zone failure, theoretical SLA jumps from ~2.5 nines to ~4 nines, and rollout includes rollbacks and a lesson about controller‑manager timing causing a transient state inconsistency.
AI-written summary · about
chapters (15)
- 0:00Control plane migration intro
- 1:03Kubernetes at Databricks
- 4:08Single-node limits
- 6:19Highly available control plane architecture
- 9:31HA failure handling
- 13:23Migration goals
- 16:32Three-phase process
- 18:18Preparing cluster state
- 21:07Bootstrapping HA control plane
- 23:17Reopening traffic
- 24:21Migration outage lessons
- 27:12Day two HA operations
- 30:00Monitoring API server
- 32:51Meeting migration needs
- 34:55Q&A
related talks
30mBuilding an Active-Active HA Kubernetes Control Plane Clu...
30mScaling Kubernetes to Thousands of Nodes Across Multiple Clusters, Calmly
34mScaling Kubernetes up to 1,000 Control Planes - Dario Tranchitella
35mClusters as Cattle: How to Seamlessly Migrate Apps Across Kubernetes Clusters
37mManaging and Running Multiple Kubernetes Clusters in Hybrid Setups
23mDancing With the Pods: Live Migration of a Database Fleet While Serving Millions of Queries
35mIntelligent Load Balancing in Kubernetes
31mHow to Migrate 700 Kubernetes Clusters to Cluster API with Zero D...