Stateful Workloads at Scale – Extending Kubernetes for Running OLAP Databases as a Service.
summary
Runs ClickHouse for several thousand customers on Kubernetes using MultiSTS (one StatefulSet per replica) and a Make-Before-Break process that creates new replicas, waits for potentially hours of ClickHouse bootstrap, then removes old ones, preserving full capacity and zero downtime. Orchestration uses a migration controller and Temporal, solves zone scheduling and PVC timing, removes persistent-PVC dependency, and relies on snapshot restore backups — a year-long migration tail but successful in production.
AI-written summary · about
chapters (9)
- 0:00Introduction and agenda
- 1:55Stateful workloads on K8s
- 6:14Limitations of default tools
- 10:24Extending Kubernetes
- 11:05The MultiSTS concept
- 16:48Make-before-break strategy
- 21:09Rolling out migrations
- 22:58Lessons and takeaways
- 26:28Conclusion
related talks
41mData warehouse and Kubernetes: Lessons from ClickHouse Operator
35mWhen StatefulSets are not enough
23mDancing With the Pods: Live Migration of a Database Fleet While Serving Millions of Queries
35mStateful Apps On Kubernetes - Bring Them On!
55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using Kubernetes
17mBuilding Real-Time Data Lake with ClickHouse
25mPostgres on Kubernetes for the Reluctant DBA
25mStatefulsets on Kubernetes: The Guardians of Your Data Galaxy