Don't gamble when it comes to reliability
summary
A single mistyped command changes firewall rules from insert into firewall group all services to insert into all services the firewall service causing every service to default to deny and triggering widespread traffic loss; the attempted fix runs a Redis pop across a shared app cluster so the bad rule lands on many machines and even the dashboards go down. Shows concrete mitigations: reduce blast radius, separate control plane from data hosts, validate destructive commands, and automate rollbacks.
AI-written summary · about
related talks
14mHow to Not-Mess-Up Production
41mRestarts and rollbacks don't fix everything
27mSoftware won't make noise when it breaks
22mDon’t wait for an outage to improve your reliability
01h 03mWhispers in the Chaos: Monitoring Weak Signals
18mLimiting the Blast Radius: Preventing Outages from Bad Configs with Datacenter-Scale Health Checks
25mWar Room Warrior: How to manage war room situations
20mIt passed auth, then production caught fire.