Service Incident Deep Dive: Technical Overview & Learnings
summary
A July 2021 Akamai incident unassigns traffic for about an hour for a subset of customers after a semantically wrong change to a flexible assignment feature reaches all edge clusters. A buggy configuration safety Stager uses “wait for good” logic plus a reporting timeout, so safety checks clear and the bad config propagates; small-stage serve failed DNS responses alerts don’t tie to rollout stages, and UI tooling slows rollback. Remediations add governance, tighten checks, and speed rollback.
AI-written summary · about
chapters (10)
- 0:00Akamai and platform reliability
- 2:10July 2021 incident overview
- 4:36Akamai Edge platform tech
- 5:39Bad config rejection failure
- 8:50Staged rollout flaws
- 11:39Tooling dependency issues
- 12:44Alert and rollout coupling
- 14:06Rollback delays and UI limits
- 14:48Key takeaways
- 15:49Internet reliability outlook
related talks
02mOpening Remarks
32mKeynote: Incident Investigation at Planetary Scale
30mIncident Groundhog Day
30mUsing incidents to level-up your teams
25mWe're Still Down: A Metastable Failure Tale
40mWhen Incidents Fix Themselves: AI SRE in action
22mWhy is it always DNS, TLS, and bad configs?
31mThe Art of Turning Incidents into Opportunities