A Transformation Story: How We Failed at 20,000+ RPS and What We Learned
summary
A postmortem-style case study of a system that suddenly faced over 20,000 RPS at a critical moment, describing the operational and technical decisions taken under crisis to limit damage and restore service. Covers incident management steps and infrastructure-level responses used to handle the high load (scaling, mitigation, and recovery actions).
AI-written summary · about
chapters (21)
- 0:00Introduction
- 3:41Load expectations and architecture
- 6:34Olympiad launch and crash
- 9:26Crisis response and communication
- 12:23Bottlenecks in infrastructure
- 14:36Scaling pods and performance
- 16:03Monitoring setup
- 18:22Lessons learned and changes
- 24:25Technical debt management
- 27:13MVP misconceptions
- 29:45Q&A
- 31:09Top five actions
- 33:08Caching strategies
- 35:13Monitoring and logging
- 37:05Legacy and team turnover
- 38:10Backlog management
- 40:20Time loop thought experiment
- 42:05Learning from mistakes
- 42:47Database vs code bottlenecks
- 44:10Business attitude shift
- 45:16Q&A
related talks
30mKeeping Your DevOps Transformation from Crushing Your Ops Capacity
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
45mToward a Culture Where Everyone Is Responsible for Outages
55mDevOps @Scale (Greek Tragedy in 3 Acts)
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
47mThe Story of an Incident: How to Paralyze Work for 20k Employees
45mFailure Happens: Improving Incident Response in Large-Scale Organizations
37m5 Years of Rails Scaling to 80k RPS