Handling Risky Business: Cluster Upgrades
Talkat KubeCon + CloudNativeCon North America 2019 · on Thursday, 2019-11-21· at 10:55· Ballroom Sec 20CD - San Diego Convention Center, Upper Level·
summary
Company builds a small Golang controller called k8srotator that runs as a single-pod custom controller on each cluster, watches ConfigMap triggers, flips a finite-state machine and updates ASG launch configs to new AMIs, drains nodes and boots replacements one node at a time. It cordons/drains, waits for autoscaler and bootstrap, implicitly freezes on API or quorum failures and alerts, and is in production at Lyft, reducing operator load and successfully standardizing upgrades across multiple clusters.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 1:00Operational challenges
- 3:24Goals for ideal tooling
- 6:27Building Cates Rotator
- 8:30Tool architecture and design
- 12:03Node lifecycle and workflow
- 15:58Iterating and handling errors
- 18:23Status and future goals
- 20:24Q&A session
related talks
44mPreventing Controller Sprawl From Taking Down Your Cluster - When a Scalable Pattern Stops Being Scalable
41mScaling Reliability Engineering with Tools
28mScaling in Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes
28mDesigning for Failure: How to Manage Thousands of Hosts Through Automation
50mOps issues
46mEnd-Users Won't Even Notice, Painless Kubernetes Upgrades
35mExtending Deployment for Internet Financial Mission-Critical Scenarios
27mSecure Multicluster Istio Configuration Using GitOps...