Delete This: Decommissioning Servers at Scale
summary
Facebook runs decommission at scale across 12→14 sites and tens of thousands of servers and replaces a months‑long, spreadsheeted process with a UI plus a job framework that spawns migration handler jobs, a task state machine tied to internal tickets, and an automated glue factory re-provisioner. Drains range from minutes to weeks; one cache bug sent live machines to the glue factory and killed them, so a new returned asset state and ticketing fail-open protect against that. After a year of rollout, automation cuts human intervention, errors and time, and enables rebalancing and DR migrations.
AI-written summary · about
chapters (5)
- 0:00Start
- 2:19A brief history of decoms
- 5:51First steps
- 9:47outages, incidents
- 13:34Service migrations beyond decoms
related talks
25mDelete This: Decommissioning Servers at Scale
19mScaling Back-End Deployment
41m"A Day in the Life of Facebook Operations"
34mZero Downtime Database Migrations and Deployments
47mDeploying Like a Boss: From 2 Servers to 20,000
25mMass migrations to the cloud
29mTaking Over & Managing Large Messy Systems
40mA Customer Service Approach to SRE