Case Study: Lessons Learned from Our First Worldwide Outage
summary
A dormant IncapRules double-quote parser bug triggers crashes on every proxy because the centralized Rev Lite leader repo replicates the malformed rule into each server’s local repo, causing a crash loop; operators revert the rule to recover. Fixes introduce a Rev Lite sandbox (sandbox repo + sandbox proxy + repo server) that quarantines and validates changes before promoting them to production, plus copy-on-write Btrfs snapshots for fast rollback and a P0/P1/P2 reliability framework to prioritize testing and architecture.
AI-written summary · about
chapters (6)
- 0:00System architecture overview
- 5:57The critical incident
- 8:21Safety mechanisms and recovery
- 13:14Post-mortem and reliability
- 18:33Implementing the sandbox
- 24:42Snapshotting and final thoughts
related talks
36mUnintended consequences of well-meaning changes
30mA Series of Unfortunate Bugs
41mThe CVE Blind Spot: Defeating "Hidden EOLs" and Repo Jacking with Engineering Triage & Code Diet
35mRedundancy Driven Development - Navigating disruptive codebase changes.
42mObject reorientation
46mEnforcing Architecture in Legacy Systems
31mInternal Background Agent Systems: Architecture and Rollout
28mMaking the Impossible Impossible: Improving Reliability by...