The Many Ways of launching AWS spot instances
summary
Red Hat CKI uses AWS spot instances for kernel CI, then faces mass job failures from UnfulfillableCapacity and InsufficientInstanceCapacity errors when a single instance type and AZ run out. Investigation finds five EC2 spot APIs; the correct fix is to use launch templates with multiple instance types and AZs via CreateFleet (or Auto Scaling) and the capacity-optimized allocation strategy. Implements API change in the runner plumbing, restoring reliability with similar cost but far fewer wasted retries.
AI-written summary · about
chapters (19)
- 0:00CKI and spot instances intro
- 1:01CKI team and CI pipelines
- 3:33Why CKI uses spot instances
- 8:29Switching to AWS spot instances
- 9:57Spot failures in December
- 12:07Investigating spot errors
- 14:12AWS Cloud and hyperscalers
- 16:00AWS instance types and pricing
- 19:35Reading AWS docs for solutions
- 21:20Five ways to launch instances
- 23:50Why createFleet is better
- 25:55createFleet in GitLab Runner
- 26:51Q&A: Cost vs reliability
- 28:11Q&A: Region and data limits
- 29:03Q&A: Kernel and RPM testing
- 30:09Q&A: Job capacity limits
- 31:16Q&A: Netflix demand spikes
- 33:02Q&A: Multi-region and AZ plans
- 34:34Q&A: Lessons and future
related talks
43mSpot Instances
42mAWS Spot instances: huge savings (or huge risks?)
43mRunning CloudBees Core on EKS: Best Practices & Amazon EC2 Spot Instances
31mEKS Cost Optimization with EC2 Spot and A1 Instances
32mBlending Spot and On-Demand Instances and Optimizing Costs
21mHow you can scale up with Amazon EC2 for up to 90% less
47mReliable, secure, and efficient cloud operations
34mSelecting and optimizing Amazon EC2 instances