Ooops... I Deleted the Whole Production Table. What Did I Learn From It?
summary
At 6pm a mistaken rm removes an entire production table with no backup; the team builds a modular backfill pipeline that bypasses CDC, scans the source and streams rows into the target, and restores service in under half a day (about six hours to get machines back). Key takeaways: immediately stop further damage, keep data-recovery playbooks and separation of logic so emergency code is usable, and code as if you will delete production.
AI-written summary · about
related talks
21mHandling a Major Outage Transparently
23mYou put out the fire on production, what’s next?
22mAfter the dust has settled: learning from incidents
39mWhat a full production crash taught us
47mEfficiency up, failed releases down; compliant production data for your test and dev environments
10mOptimizing incident response
43mDisaster Non-Recovery
41mRestarts and rollbacks don't fix everything