Software won't make noise when it breaks
summary
Provides concrete failure stories and fixes: a cloud operator deleted 443 rule in a security group and missed the outage because external checks never hit internal monitors; a data-center node lacked a fix because it missed an ansible run causing a shard to fail; a distributed lock breaks because of clock drift/ntp missing that makes short TTLs expire; remediation is not more monitoring but enforced infrastructure as code, osquery/boot-time config checks, simple NTP validation, and a Last9 knowledge graph that maps component relationships to predict cascade impact.
AI-written summary · about
chapters (11)
- 0:00SRE and pager culture
- 1:01Modern systems break silently
- 2:25Hidden costs of failures
- 3:06SRE principles for incidents
- 5:13Outage: Security group deletion
- 9:44Outage: Mount command failure
- 15:41Outage: Distributed lock drift
- 21:39First principles over tools
- 24:31Learning from failures
- 25:55Knowledge graph for reliability
- 27:19Q&A
related talks
01h 03mWhispers in the Chaos: Monitoring Weak Signals
16mEngineering Failure-Resilient Systems
15mMaintaining reliable systems
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
22mDon’t wait for an outage to improve your reliability
22mWhy is it always DNS, TLS, and bad configs?
41mRestarts and rollbacks don't fix everything
23mNavigating in the Dark