Surviving operational "Chernobyl"
summary
Built a DevOps competence center that supports life‑critical systems (airline comms, eCall) and enforces runbooks, documentation and regular outage drills, with specific incident lessons: 1.5M eCalls in two years and ~100 missed per hour statistics drive urgency; missed SLA from forgetting to close a ticket; UI regression (missing end‑call button) slipped past QA; 2FA outage is mitigated by disable 2FA via config after stakeholder sign‑off; Terraform runs once erased custom AWS config; noisy test servers send 50,000 GETs causing Docker reboot loops; Jenkins instability prompts a dedicated instance after 21 days of Jenkins outages.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 2:50DevOps sub plug-in approach
- 8:30Preparing for production incidents
- 13:08Real production cases
- 46:37Summary
- 48:22Q&A
related talks
42mSurviving operational "Chernobyl"
48mSurviving Operational 'Chernobyl'
47mHow did I Help to Save the Lives of More Than 1.5 Million People?
55mDevOps @Scale (Greek Tragedy in 3 Acts)
52mThree years of breaking things to make them better
30mDevOps: Supporting aircraft from hanger to take-off
48mDevops@Scale
50mDevOps Project Failures - Lessons Learned and Strategies for Recovery