Whispers in Chaos: Searching for Weak Signals in Incidents
summary
Research on an Etsy Cyber Monday outage shows slow personalized pages caused by CDN cache misses triggered by an HTTP 400 from a closed store referenced in a blog widget; investigators stitch IRC messages and access logs into per-engineer timelines using process tracing and find three operational heuristics: what changed, a “go wide then deep” search pattern, and convergent searching driven by painful recent memories (L1 cache). Engineers often skip full test runs under pressure; postmortems succeed when they produce learning, not just remediation tickets.
AI-written summary · about
chapters (17)
- 0:00Start
- 2:58Two Brain Systems System One
- 4:08Two Problem Types "System One"
- 6:41Timelines, yadda, yadda
- 9:04Software: A Team Sport
- 10:13Change
- 11:19"Go Wide"
- 12:30Convergent Searching
- 19:26Bonus Heuristic: Testing the Fix
- 21:32Elements of Expertise
- 24:00Seeing the Invisible
- 25:00Yeah, but Malcolm Gladwell...
- 25:41Expert Performance
- 27:15Expertise in Other Crafts
- 29:38Transforming Experience into Expertise
- 34:10Maslow's SRE Hierarchy
- 34:59Just Two Questions
related talks
01h 03mWhispers in the Chaos: Monitoring Weak Signals
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
30mIncident Groundhog Day
30mOff the Beaten Path: Moving Observability Focus from Your Service to Your Customer
25m"The website's down!" Stories and lessons on keeping your website up.- 01h 03mWhispers in the Chaos: Monitoring Weak Signals
39mConnecting the dots: observability in large e-commerce environments
29mSex, Lies & Videotapes: The Truth Behind Your CDN Logs