Commas Save Lives, or at Least LinkedIn
summary
A single missing comma in JSON entry to an anti‑abuse ACL causes LinkedIn’s front door to drop to zero requests, Apache Traffic Server enters a crash loop because a missing try/catch after a version upgrade removes validation, and the nurse auto‑remediation repeatedly restarts the failing service. Detection is three minutes, mitigation takes 1 hour 51 minutes, and fixes include isolating ATS tiers, adding validation plus unit tests, a reviewed ACL change process, defined break‑glass paths, and creating a central incident management rotation.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:56Incident Management at LinkedIn
- 5:23Moving Desktops to Azure
- 7:05Incident Timeline to Mitigation
- 10:30Contributing Factors
- 13:17What Went Right?
- 14:50Too Many Cooks
- 20:39Fixing the Technology
- 22:52Incident Commander Role
- 24:43Incident Response Overhaul
- 26:56Future of Incident Response
related talks
20mHow LinkedIn Performs Maintenances at Scale
28mAI-Driven Incident Response: From Alert to Postmortem
13mEvading and crashing anti-malware solutions via data collection overloading during...
39m30 minutes of Failure
26mThe Sounds of Silence: Lessons from an 18 hour API outage
30mAPI Drift: A Hidden Crisis in MCP Servers
51mHearts of Darkness: A Spring DevOps Apocalypse
30mAppSec Timeline: Wins, Failures, Promises, and Predictions