Incidents and incident handling @ VG.no
summary
VG runs a written IMAS incident framework plus tooling: a live incident document (LID) auto-created with Slack/Google Drive links, bots to open channels and notify stakeholders, and clear roles (most importantly Incident Commander and comm/ops leads). Infrastructure defenses include BGP address failover, multiple CDNs and a varnish cache front, websocket routing and randomized client polling. Operations see DDoS daily (one peak ≈600 Gbit/s costing ~1.3M NOK), ad-delivered iOS zero‑days that forced disabling programmatic ads, and postmortems with action items after every incident.
AI-written summary · about
chapters (10)
- 0:00Introduction to VG
- 2:11Operational challenges
- 4:48Defense in layers
- 7:23Handling DDoS incidents
- 10:21Preventing self-inflicted downtime
- 13:06Mitigating the thundering herd
- 16:05Malicious ad investigations
- 22:55Incident management framework
- 28:33Automating incident response
- 34:48Culture, process, and Q&A
related talks
27mManaging Incidents
20mHow to Have an Operational Incident
19mHow to handle incidents at scale
28mHow to Have an Operational Incident (A Crash Course)
30mHow To Have An Operational Incident (a crash course)
22mSmarter Disasters: End-to-End Automation for Incidents
51mDatabase Incident Management: When Everything Goes Wrong
34mIncidents Happen