~/talk

Planning To Fail

on YouTube 42m
Talkat PHP UK Conference 2013 · 2013-02-21..02-23· [devops-and-delivery, observability]

speaker

name:Dave Gardner
links:none on file

resources

video:https://youtu.be/hHwNEuTu6rQ
slides:none on file

summary

Builds resilience by breaking things: runs around ten backend services on a couple hundred EC2 instances, sees ~30% month growth and ~1 request every 5 seconds per region, and replaces MySQL with Cassandra across three continents plus Zookeeper for coordination, ElasticSearch for search, NSQ for messaging and a CR_flake ID service. Automates fault injection with JMeter, iptables experiments and Graphite assertions, runs automated nightly failure tests, uncovers a memcache client timeout bug and fixes it so degraded paths work reliably.

AI-written summary · about

chapters (4)

  1. 0:00Start
  2. 0:44Planning for failure
  3. 6:19reliability is in danger
  4. 23:42Things fail in exotic ways

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.