How to SRE When Everything's Already on Fire
summary
Squarespace fixes a months‑long ELK outage by turning vague alerts into a single consumer‑lag / write‑rate SLI, raising SLOs from “5 min 99%” to “2 min 99.9%” and cutting shard chaos (nodes had ~2,200 shards vs a recommended ~600). Built: per‑app ingestion meters, tagged Logstash filters, Prometheus exporters, shard‑movement tracker, index precreation scripts, a query killer and Logstash blackhole. The worst outage lasts 37 hours; ongoing stability follows after the unsharding run and ICS‑style incident command.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:23A PHENOMENAL EVENING
- 3:34ELK @ SQUARESPACE
- 11:02SERVICE RELIABILITY PRINCIPLES
- 11:40THE RELIABILITY STACK
- 12:14SERVICE LEVEL OBJECTIVES
- 13:43ERROR BUDGETS ARE AWESOME
- 21:11THIS RELIABILITY STUFF ISN'T NEW
- 21:33THE INCIDENT COMMAND SYSTEM
- 22:06PROBLEMS THE ICS ADDRESSES
- 23:33OPERATIONS LEAD
- 23:54INCIDENT COMMANDER 1
- 24:25TIMELINE OF A 37-HOUR INCIDENT
- 26:24SEE THE FOREST FOR THE TREES
- 27:28THE UNSHARDENING
- 28:29KEY COMPONENTS
- 29:55LESSONS LEARNED
- 30:43REPAIR ITEMS
- 33:19PROGRESS IS INCREMENTAL
- 34:25ALERT ON WHAT MATTERS Put your users first
related talks
23mDiving through RCAs a quest for reliability
25m"The website's down!" Stories and lessons on keeping your website up.
35mSRE -- culture and strategy @ TrustingSocial
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
41mScaling Observability: From Chaos to Clarity with Open Source Tools
31mServerless Observability: a case study of SLOs
38mTransforming Production Readiness
21mSRE, what is it all about?!