How to survive a Lambda-induced data tsunami in AWS
summary
AWS logging failures trigger a six-hour outage in healthcare apps after a single log entry is duplicated 80,000×, causing Lambda functions to re-ingest and retry payloads that exceed size limits and cascade. Teams rule out cyber attack and application changes, switch observability tools to confirm duplication, then disable rogue Lambda to stop the tsunami. Post-incident forensic fixes to config and code aim to prevent recurrence and use Observability as Governance for regulatory compliance.
AI-written summary · about
chapters (21)
- 0:00Welcome and introduction
- 1:24The Thursday afternoon crisis
- 3:10Introducing the team and agenda
- 3:50The unfolding incident
- 7:05Ruling out cyber threats
- 8:10Investigating application issues
- 8:32Platform and log analysis
- 9:34Discovering log duplication
- 10:41Identifying rogue Lambda
- 11:46Root cause and resolution
- 13:09Six-hour incident timeline
- 14:32Observability as transparency
- 15:59Current observability gaps
- 17:04Regulatory push for observability
- 18:28Mapping critical business services
- 19:30Continuous compliance and audits
- 20:34Governance and change management
- 21:58AI for observability and governance
- 22:40Mindset shift and real-time dashboards
- 23:22Key takeaways and next steps
- 24:44Closing and feedback
related talks
05mTales From the AWS Crypt - A Collection of User Horror Stories
36mObserving all the Serverless Things
34m5 Things I Learnt from Causing a Cloud Provider Outage
31mOn the Importance of Visibility
26mA proactive approach to resilience in modern apps
25mHow to Fail at Serverless (without even trying)
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
30mThe last in the line: a talk about DevOps and Disasters