Mystery of the disappearing S3 Keys
summary
Monitoring pipeline ingests ~1.5–2 million S3 object events per day and shows ~350 daily failures where code sees an object but S3 reports NoSuchKey. Investigation finds S3 provides strong consistency for PUT/GET but not for COPY, so race conditions between EventBridge/SQS-triggered Lambda batches and S3 replication cause intermittent missing keys. Fix is to use the SQS–Lambda partial batch response so only failed records are retried, eliminating bulk reprocessing and resolving the rare disappearing-key errors.
AI-written summary · about
chapters (7)
- 0:00Introduction and Conference Overview
- 1:36Meet the Speakers: Simon and Ross
- 3:12The Monitoring Application Challenge
- 5:00Investigating the Unexpected Errors
- 8:26Understanding Consistency in Distributed Systems
- 11:27AWS Solutions and Final Fix
- 14:03Lessons Learned and Conclusion
related talks
40mSh... Fail Happens: Fail-aware Events Processing on AWS
53mBottlenecks in Serverless Applications
26mHow to survive a Lambda-induced data tsunami in AWS
36mObserving all the Serverless Things
37mA Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day
49mUsing serverless functions for real-time observability
32mUsing Serverless Functions for Real-time Observability
40mManaging to Your SLO Amidst Chaos