Oops. I broke the Google, now what?
summary
Runs Google authentication serving hundreds of millions of QPS across 190 services and several petabytes of storage and sometimes still causes company‑wide outages. Achieves up to 99.9999% (31.56s/yr) for critical pieces but learns from big failures in 2015 2017 2020. Uses blameless postmortems with five whys, client‑side SLIs/SLOs and cost‑based alerting, chaos testing, canary deployments at 1% with auto‑rollback, strict on‑call limits (≈ two incidents per 12h shift) and dashboards traces and playbooks.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 3:56Impact of outages on users
- 6:01Outage lifecycle explained
- 10:17Writing effective postmortems
- 14:30Shortening time to detect
- 17:17Reducing responder fatigue
- 20:30Testing production systems
- 25:51Time to repair best practices
- 30:59Extending time between failures
- 35:55Key takeaways and resources
- 36:38Q&A session
related talks
43mGoogle - Load-shedding
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
38mGoogle's DevOps Culture
44mDevSecOps: Balancing Agility, Security & Stability in a Continuous Software Business
56mGetting all the 99.99(9) you always wanted
33mThe DiRT on Chaos Engineering at Google
35mSRE 101
39m10 Learnings From Running Production Infrastructure at Google