The last in the line: a talk about DevOps and Disasters
summary
Relay42 handles ~70,000 events/sec and experiences corner-case outages where a single request pattern triggers expensive per-event Cassandra lookups and causes Java heap deaths; fixing it with deduplication and macro batching reduces pressure from about 60 million Cassandra calls at peak to near zero and takes roughly two and a half months. A later failure hits a hidden 256-table DynamoDB limit caused by leaked per-instance tables, fixed with cleanup Lambdas and saving hundreds monthly. Lessons: name incidents, keep infra as code, add SLOs, and avoid blind self-healing.
AI-written summary · about
chapters (13)
- 0:00Intro
- 1:50About Nikola
- 3:04About Relay
- 4:12Agenda
- 5:12Platform Stability
- 6:35Example App
- 7:42Tracking
- 9:11Touch of Death
- 17:11Results
- 19:22Limits
- 22:45What we didnt know
- 25:29Future plans
- 27:21QA
related talks
55mDevOps @Scale (Greek Tragedy in 3 Acts)
31mDevOps in the Cloud: Case Studies of Amazon.com teams and their resilient architectures
20mDevOpsDays 2026 Ignites Day 1
42mPlanning To Fail
51mHearts of Darkness: A Spring DevOps Apocalypse
31mEscaping operations hell
40mThe Art of Embracing Failure at Scale
26mA proactive approach to resilience in modern apps