Embrace Fleet Reboots and Make Them Boring
summary
Cloudflare standardizes monthly reboots across 300 edge cities by building an automation stack: Quicksilver for coordinated state, a Bing reboot server queueing nodes, per-role reboot limits and dynamic maintenance windows. A prior kernel+BIOS bug caused ~20 crashes/day and drove automation; emergency kernel rollout reaches 90% in 72 hours using staged dogfooding/canary tiers. Reboots run on a randomized 29-day cycle, include preflight health checks, a reboot-rescuer for failed nodes, and cut manual toil (≈500 person-hours per region) to near zero.
AI-written summary · about
chapters (10)
- 0:00Fleet reboots introduction
- 3:48Edge vs core infrastructure
- 8:07Why servers reboot
- 11:47Chaotic 2018 reboots
- 13:10Automating with Rudy MOG
- 18:07Bing Reboot Server
- 21:03Emergency kernel upgrade
- 24:44Reboot resilience & health checks
- 27:13Quarterly releases
- 30:47Key takeaways
related talks
15mKeeping a Hypervisor Fleet Up to Date with Minimal Customer Disruption
49mLong-Term Maintenance, or How to (Mis-)Manage Embedded Systems for 10+ Years
22mKeynote: Cloud Native Security: Rotate, Repair, Repave
41mHave You Tried Turning It off and Turning It on Again?
52mThree years of breaking things to make them better
22mUpdate Strategies for the Edge
31mRadical Ideas Enterprises Can Learn From The Cloud
28mDesigning for Failure: How to Manage Thousands of Hosts Through Automation