"The website's down!" Stories and lessons on keeping your website up.
summary
A high‑traffic site suffers three outage days in one week with 21 outages in a single afternoon caused by Redis session spikes, prompting a rapid switch to an in‑memory store and then to DynamoDB for sessions. Centralized logging via Elasticsearch replaces scattered log files, disk sizing forces node and retention tradeoffs, and CDN caching prevents about 400,000 pages cached from becoming errors. New designs serve static homepage from S3, use preaggregated DynamoDB for details, and add circuit breakers, timeouts and tuned Pingdom/New Relic/OpsGenie alerts.
AI-written summary · about
chapters (7)
- 0:00Start
- 1:39Redis Store for Session Data
- 4:52Step back and evaluate
- 15:45New Design
- 18:24Circuit Breakers
- 19:36Timeout
- 22:39Quick Recap
related talks
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
27mFAIL: The Best Ways To Bring Down Your Website
10mGreat Caching Disasters!
34m"The Upside of Downtime: How to Turn a Disaster Into an Opportunity"
33mFrontend Resilience
40mHow to SRE When Everything's Already on Fire
40mWhispers in Chaos: Searching for Weak Signals in Incidents
26mThe tools and techniques behind resilient systems