~/talk

What to do when complex systems fail: A guide to chaos management

on YouTube 45m
Talkat Full Stack Fest 2017 · 2017-09-04..09-08· [devops-and-delivery, observability]

speaker

name:Aish Raj Dahal
links:none on file

resources

video:https://youtu.be/d4CyX4QaNL8
slides:none on file

summary

Title gives specifics: a production log aggregator ran out of disk space causing reports to fail and a two‑hour escalation where 16 engineers on a Friday night all chase the same last Git commit. Framework maps to NIMS roles (incident commander, deputy, scribe, customer liaison, SMEs), ties outages to business metrics (PagerDuty uses outgoing notifications/sec), and recommends prework like Failure Friday simulations plus automation and chatops — e.g. a chatops command to start a major incident that pages roles so fixes (cron, monitoring) get built and human time is spared.

AI-written summary · about

chapters (38)

  1. 0:00Introduction
  2. 1:07What are complex systems
  3. 2:25How complex systems fail
  4. 3:51Structure of the talk
  5. 4:54Chapter 1 Failure
  6. 5:52Chapter 1 Incident
  7. 7:18Chapter 2 Dark and Stormy Night
  8. 8:25Chapter 3 The Soup
  9. 10:29Chapter 4 The Morning After
  10. 12:20Chapter 5 Lessons Learned
  11. 16:23Chapter 7 National Incident Management Systems
  12. 17:25Chapter 8 Single Responsibility Principle
  13. 18:17Chapter 9 Incident Calls
  14. 20:10Chapter 11 Incident Commander
  15. 21:37Chapter 12 Incident Commander
  16. 22:24Chapter 13 Divide Conquer
  17. 23:06Chapter 14 Communicate Effectively
  18. 24:08Chapter 15 Avoid the bystander effect
  19. 25:02Chapter 16 Reduce scope
  20. 26:10Chapter 17 Maintain order
  21. 26:30Chapter 18 The Deputy
  22. 27:50Chapter 19 The Deputy
  23. 28:52Chapter 20 The Scribe
  24. 29:07Chapter 21 The Scribe
  25. 30:24Chapter 22 The Customer Liaison
  26. 31:51Chapter 23 The Customer Liaison
  27. 32:38Chapter 24 The Customer Liaison
  28. 33:02Chapter 25 The Incident Commander
  29. 33:38Chapter 26 The Morning After One
  30. 34:37Chapter 27 Reliability
  31. 34:55Chapter 28 Role of Incident Commander
  32. 36:02Chapter 29 Role of Incident Commander
  33. 36:51Chapter 30 Wartime vs Peacetime Expectations
  34. 37:59Chapter 31 The Reveal
  35. 38:15Chapter 32 The Framework
  36. 38:37Chapter 33 Develop on call empathy
  37. 39:33Chapter 34 People are most valuable asset
  38. 41:04Chapter 35 How do you manage late events

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.