What I Wish I Knew Before Choosing Spot Instances
summary
Runs Loki, Grafana, Tempo and Mimir on Kubernetes as 8,000 pods across 330 nodes and saves 60–90% on compute by using spot instances, but evictions are unreliable (Azure “30s” notice often doesn’t happen), causing write‑ahead‑log corruption, HTTP timeouts, anti‑affinity/overprovision tradeoffs and hash‑ring quorum failures. Built a node‑evictions Grafana dashboard and a custom hash‑ring monitor to detect and remediate failures; still hit rare but catastrophic eviction storms: 280 nodes in 11 minutes. Result: move most workloads off spot; 70% of reliability incidents trace back to spot instability.
AI-written summary · about
chapters (17)
- 0:00Spot instances overview
- 1:34Chaos from spot instances
- 3:37Eviction architecture
- 5:42Eviction risks
- 7:27Support challenges
- 8:30Anti-affinity strategies
- 10:17Timing-based evictions
- 11:41Multi-zone quorum issues
- 14:45Complex eviction interactions
- 16:06High churn risks
- 17:32On-restart errors
- 19:16Node reliability
- 21:02Telemetry for evictions
- 22:46Eviction rates by zone
- 23:50Provisioning bottlenecks
- 24:10Eviction storms
- 25:57Lessons learned
related talks
21mLessons Learned Running GKE Clusters on Spot Instances
42mAWS Spot instances: huge savings (or huge risks?)
33mKubernetes cost optimization - How to actually do it
34mKubernetes: Reducing Infrastructure cost and complexity
47mReliable, secure, and efficient cloud operations
38mBeyond Kubernetes: Adapting To Spe...
43mSpot Instances
14mCut Costs, Not Performance: Implement Kubernetes Cost Optimization Strategies