When Your SaaS Provider Goes out of Business – Lessons from...
summary
On 2 Feb 2023 a critical URL‑filter vendor folds, risking service for 100 customers, 300k daily users across 3,500 sites; response splits into three parallel tracks: sustain operations, communicate, replace vendor. A one‑week hack builds a local URL filter feed with an index of 100,000 blocked domains, deployed via the existing telemetry pipeline to all gateways; vendor replacement takes seven months, first customer after four. Key specifics: fail‑open behavior, license validation that breaks restarts, single Slack channel, web portal + phone outreach.
AI-written summary · about
chapters (23)
- 0:00SaaS provider crisis intro
- 1:21Open Systems context
- 3:11Crisis response setup
- 4:37Crisis response plan
- 6:23First actions and sitrep
- 7:46Communication strategy
- 9:09Technical system assessment
- 11:56Failure modes and risks
- 13:44Support and telemetry
- 16:32Creating runbooks for operations
- 17:16Vendor replacement begins
- 18:00Local URL filter
- 20:34First week progress
- 21:57Sitrep cycles and phases
- 23:22Sustained operations
- 24:08New vendor integration
- 25:55Customer migration
- 27:20System design flaws
- 28:28Key tools and processes
- 30:58Crisis preparedness
- 31:40Comm and execution challenges
- 33:29Response process and intent
- 34:10Key takeaways
related talks
22mSemi-Advanced Threats vs the Semi-Critical Infrastructure
17mFive things in tech that matter and we have to make work
42mCatastrophes and Atrophies
50mSoftware Supply Chain: Panel on Threat Intel, Trends, Mitigation Strategies
15mBol.com and the Accelerated Legacy Challenge
31mThe internet is broken: breaking down the software supply chain
18mSurvival of the fittest: The Digital Evolution
27mLessons Learned From Architecting the Highest-scale Operational Systems in the World