Deployments, Downtime, Unexpected Fires: an SRE Survival Story
summary
GCP object replication fails during a data center fire with GCS dual-region 99.9%/12h guarantees, prompting adoption of turbo replication under 15 minutes; SSL outage traces to a missing intermediate certificate and expired wildcards; logging gaps delay recovery until centralized Elasticsearch or Datadog ingestion is enforced; Terraform custom modules get replaced by registry modules and moved into Terraform in CI/CD with vetting tools like viz; alerts, disk watermarks and certificate checks become mandatory predeploy steps.
AI-written summary · about
chapters (9)
- 0:00Introduction to Infrastructure Deployment
- 0:11Meet Prade Gadi: My DevOps Journey
- 0:46Real-World SRE Failures and Lessons Learned
- 1:00Data Center Incident: The Importance of Cloud Provider Terms
- 2:34SSL Certificates: Common Issues and Solutions
- 4:44Logging and Monitoring: Best Practices
- 7:12Terraform: Recommendations and Best Practices
- 9:05The Essence of SRE: Resilience and Adaptability
- 10:47Conclusion: Embracing the Unexpected
related talks
19mPreparing for disaster: going Multi-Region
47mBuilding Resilience
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
27mAutomating for Failure: Disaster Recovery Testing with Terraform & Gremlin
27mFailover Without Fear: A Practical Guide to Active-Active...
23mDiving through RCAs a quest for reliability
44mDevSecOps: Balancing Agility, Security & Stability in a Continuous Software Business
21mHow to CTF Infra - Beyond the challenges and flags