Auto Operation of Hundreds of K8s Clusters in Yahoo!
summary
Yahoo Japan runs an on‑prem Kubernetes‑as‑a‑Service built on OpenStack Magnum with custom CRDs and controllers that create machines and clusters, perform automatic healing via OpenStack API calls, and push automatic rolling upgrades with claimed zero downtime. The system manages about 400 clusters and ~4,000 VMs with a ~10‑person team, bundles Prometheus and add‑ons, and applies urgent patches across all clusters within ~24 hours, reducing operational overhead.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:51Agenda
- 1:37Welcome
- 3:03Yahoo Japan
- 4:23Team Management
- 8:47Machine Management
- 9:32Clusters
- 14:11Automatic Recovery
- 15:50Updating Custom Resources
- 21:41New Functions
- 23:22Summary
- 24:18Future
- 28:19Tim Allen
- 28:50Dava
- 32:40Version Upgrade
related talks
35mAuto Operation of Hundreds of K8s Clusters in Yahoo! Japan
33mRunning Istio and Kubernetes On-prem at Yahoo Scale
36m101 Ways to “Break and Recover” Kubernetes Cluster
40mLarge-Scale K8s Cluster Operation and Management
32mDon't Start With 500 Clusters: Patterns in Scaling Multi-Cluster Kubernetes…
39mMulti-Cluster Ops in a Hybrid World
03mFrom 5 to 1,300+ Clusters: Declarative Scaling for Private Kubernetes
23mFearless automation Deploying OpenStack on top of Kubernetes