Fast, Available, Catastrophically Failing? Safely Avoiding Behavioral Incidents in Complex Production Systems
Talkat SREcon19 Europe/Middle East/Africa · on Wednesday, 2019-10-02· at 16:45· Track 2: The Liffey A·
summary
Argues that model and data services fail as behavioral outages where bad data, not code, causes catastrophic user-visible behavior such as the high‑profile false obituary example. Recommends concrete controls: testing in prod, progressive delivery via feature flags and canary percentages, and using error budgets to buy chaos experiments. Gives specific failure modes — stale training pipelines, mislabeled training data that makes models ignore inputs, and silent default fallbacks — and says reproducible data versioning is required to detect and reverse them.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:05Outline
- 3:22Complexity and Failure
- 6:39Tools and Vocabulary
- 12:17Audacity
- 21:45Conventional Incidents
- 24:14Examples
related talks
22mFailing Over without Falling Over
55mGoing Resilient
31mDesigning for failure
36mDesigning for failure - architecting fault-tolerant system
10mBuilding an 'Oops-Less Operation' at Scale
33mChaos engineering for the rest of us
29mBuilding Resilient Systems
41mOnce Upon a Vulnerability: Building Resilience in a Chaotic World