Lessons Learned Running GKE Clusters on Spot Instances
summary
Non‑prod clusters move to Spot for up to ~90% discounts and encounter two surprises: GKE gives only 25s SIGTERM / 5s kubelet for preemption so apps must finish quickly, and Managed Instance Groups sometimes replace a VM but reattach the same node name so kubelet restarts cause confusing errors like configMap mount failures and dozens of failed pods while overall capacity is fine. Fixes: use balanced autoscaler or headroom, deploy descheduler, enforce startup probes, replicate across zones and set SLOs/PDBs to prove availability.
AI-written summary · about
chapters (11)
- 0:00Running GKE on Spot Instances
- 1:03Spot Instances trade-offs
- 2:27Non-production on Spot Instances
- 3:53Capacity management for Spot
- 6:49Workloads suitable for Spot
- 9:00Spot preemption challenges in GKE
- 13:37Edge cases in GKE Spot handling
- 15:44Preemption impact on developers
- 17:29Mitigating Spot Instance issues
- 19:32Best practices for Spot workloads
- 20:39Q&A
related talks
36mCost-effective Compute Clusters with Spot and Pre-emptible Instances
33mThe Big Kubernetes Mistake Paying More Than Necessary
27mWhat I Wish I Knew Before Choosing Spot Instances
26mScaling K8s Nodes Without Breaking the Bank or Your Sanity
29mRunning Kubernetes on 'small scale' clusters - lessons learned
33mKubernetes cost optimization - How to actually do it
42mAWS Spot instances: huge savings (or huge risks?)
27mScaling K8s Clusters Without Losing Your Mind or Money