A Tale of Two System Crashes: Innovation and Accidental Chaos
summary
Flatiron Health builds an internal Flask job runner called Crane and adds AWS Auto Scaling to create AutoCrane, which turns a static 20‑worker EC2 fleet into a variable pool that grows to ~150 workers and has launched 32,500 instances since rollout. A bug (forgotten instance protection) causes lost jobs and forced a rollback; worker polling every 10–20s plus an 8.1 MB JSON state file overloads the primary and breaks the UI. AutoCrane works but exposes upstream limits (git server CPU spikes from mass git fetches), so fixes include capped scaleups, faster instance boot, and better cross‑team communication.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 8:11Problem with static crane fleet
- 13:24Designing auto crane solution
- 21:50First unexpected outage
- 25:41Fixing git server dependency
- 26:44Auto crane exposes broader issue
- 29:33Key lessons learned
- 32:43Success and impact of auto crane
related talks
42mChaos Engineering: Why Breaking Things Should be Practiced
09mI Run a Fleet of AI Agents Across Three Machines. Here's What Broke.
29mWe accidentally created a Cloud on our IBM Cloud
27mAutomation First, Regrets Later
42mPlanning To Fail
25mWe accidentally created a Serverless Application
55mThe Troubles of Automating “all the Things”
47mDeploying Like a Boss: From 2 Servers to 20,000