How Our Security Requirements Turned Us into Accidental Chaos Engineers
summary
Built a Lifespan Management system that enforces a seven-day VM lifespan using AWS Auto Scaling Groups, a local Terminator graceful-shutdown agent, and a stateless cron that terminate-first with ASG. It reduces exposure to CVEs but reveals failure modes: a convenient maintenance-mode health check falsely passes, a different terminate API call bypasses the Terminator and kills Kubernetes workers, and there is no cancel path once termination starts. Fixes include touch termination (invoke terminator before API call), explicit service health checks, and safer onboarding.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:49Topics / Spoilers
- 1:19Terminology
- 1:38Instance Age
- 2:37What's wrong with old instances?
- 4:25CVE Patch Released
- 5:09Old Instances are bad
- 6:06What is an auto-scaling group?
- 6:26What is a terminator?
- 6:56Lifespan Manager
- 7:20Terminate First vs. Launch First
- 10:03A Year Long Journey
- 11:21How did this happen?
- 12:12How NOT to health check
- 13:01Explicit Expectations
- 13:30Terminator Recap
- 14:33RIP Kubernetes Workers
- 15:43Heartbeat Options
- 17:28Two Touch Termination
- 17:52The Blackhole Scenario
- 21:37Turning it on
- 22:08Who does what?
- 22:48False Alarm
- 23:20Self-service Meltdown
- 26:44Takeaway
related talks
29mHow Security Requirements Turned Us into Accidental Chaos Engineers
22mSecurity vs Dev Experience Threat … or Opportunity
57mChaos Engineering Primer
35mKubernetes Security - Challenge and Opportunity?
26mBoost your security and resiliency in Kubernetes
27mExecuting Chaos Engineering in Production at a Critical Financial Institution
33mImprove resilience with automated chaos engineering
46mDesigning a System for Chaos