5 Things I Learnt from Causing a Cloud Provider Outage
summary
A team causes an outage across Europe by overloading a cloud data lake and loses access to petabytes needed for threat hunting. The rebuild uses Lambda‑style FaaS + scatter‑gather across thousands of tables, Kubernetes operators, Kafka/SNS messaging and a workflow orchestrator like Apache Airflow. Concrete fixes: store job metadata in a document DB, make tasks idempotent, alert on user‑facing retries not CPU, and give the cloud provider exact account/resource IDs.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 10:16Breaking the wrong block first
- 13:27Build with failure in mind
- 17:42User experience for tech users
- 23:22Optimizing complex queries
- 25:49Observability beyond metrics
- 29:06Designing for reliability
- 32:20Resources and closing
related talks
26mHow to survive a Lambda-induced data tsunami in AWS
29mWe accidentally created a Cloud on our IBM Cloud
53mResiliency and Availability Design Patterns for the Cloud
34mLessons Learned from Five Years of Multi-Cloud
36mCloudy With a Chance Of Chaos: Verifying the Resiliency Of Cloud-Native Applications
22mBattles Wounds from a Cloud SRE.. And a few lessons
40mAvoiding Pain When Operating in the Cloud
30mThe last in the line: a talk about DevOps and Disasters