The Story of an Incident: How to Paralyze Work for 20k Employees
summary
Postmortem of an incident in an internal Yandex Tracker deployment: how their incident-management pipeline detected and responded to a failure that paralyzed work for ~20,000 employees. Covers monitoring and detection techniques, response procedures, and operational practices used to investigate and remediate the outage.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 12:10What is Tetris in tracker
- 13:33Elasticsearch 6 problems
- 20:26Incident start and OpenSearch switch
- 25:27HTTP-pool locks and service downtime
- 27:53Recovery and incident lessons
- 33:15Tetris evolution and outcomes
- 33:57Q&A
related talks
11mUsing incidents to level-up your teams
22mAfter the dust has settled: learning from incidents
45mFailure Happens: Improving Incident Response in Large-Scale Organizations
19mHow to handle incidents at scale
47mA Transformation Story: How We Failed at 20,000+ RPS and What We Learned
27mLessons from Incident Management and Postmortems at Atlassian
46mKeynote Presentation Survive Your Nightmare Security Incident and Thrive Again
37mBlameless Incidents: Learning from Failure at Scale