What Breaks Our Systems: A Taxonomy of Black Swans
summary
Catalog of real failure modes: hidden resource caps like ext3 2TB limit that force a dump-and-reimport to ext4, logical counters such as Postgres xid wrap that stop writes, provider limits like AWS connection tracking, and buffer caps (zookeeper 64K pipe). Failures spread via saturated queues (Redis client retrying 5×), thundering herds from client failover or GitHub reloads, automation mistakes (a regex wiped global CDN), and malware ( NotPetya hit Maersk $300M ). Defenses: full-load writes testing, timeouts + circuit breakers, exponential backoff, strict automation limits, global logging and network segmentation.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:13What is a Black Swan?
- 2:24Black swans can become routine non-incidents
- 3:41Some subspecies of black swan
- 9:07Defence: load and capacity testing
- 10:43HostedGraphite February 2018
- 14:50Where does coordinated demand come from?
- 22:12Defence: smaller blast radius
- 23:14Dependency loops
- 26:18Defence: incident management process
- 26:52Defence communication
- 27:40Defence: priorities and budgets
related talks
30mWhat Breaks Our Systems: A Taxonomy of Black Swans
34mDistributed Systems - What Can Go Wrong and How to Get It Right
29mHorrors of Distributed Systems
29mBuilding Resilient Systems
30mFast, Available, Catastrophically Failing? Safely Avoiding Behavioral Incidents in Complex Production Systems
36mDesigning for failure - architecting fault-tolerant system
31mDesigning for failure
10mBuilding an 'Oops-Less Operation' at Scale