Autonomous Automation: How Cloudflare Handles Server Diagnostics...
summary
Cloudflare runs an autonomous system called Phoenix that discovers servers in repair across 285+ PoPs in 100 countries, boots a custom Linux image to run hardware tests, evaluates results, reprovisions OK nodes and leaves failures for a decoupled RMA workflow. It consumes Prometheus data for alerts and historical trends, coordinates with other automations, and enforces an error budget of two recoveries per SRE shift across three time zones (budget resets daily) to stop recoveries when failures spike.
AI-written summary · about
chapters (8)
- 0:00Intro
- 1:29Automation and Autonomy: What's the Difference?
- 5:37The Cloudflare Network
- 7:44Automating Server Diagnostics and Recovery
- 11:55Infusing Intelligence: Consuming data from Prometheus
- 13:28Infusing Intelligence: Consuming data from other systems
- 15:18Situational Awareness Achieved
- 15:47Error Budgets for Automated Systems
related talks
14mFrom Dashboard to Defense: Automating Resilience at
05mBuilding Self-Adaptive Autonomous Infrastructure
18mAlert Analytics
42mPlanning To Fail
40mWhen Incidents Fix Themselves: AI SRE in action
26mSelf Healing Systems
42mHow to stay alive even when others go down:...
19mYou Can't Stop Fires with an Ambulance