Chaos Engr. and Service Ownership at Enterprise Scale
summary
Salesforce migrates to Hyperforce and builds a self-service chaos platform that runs privileged attacks across about 80 AWS accounts and orchestrates faults across multi-tenant Kubernetes clusters without granting shell access. Platform integrates with SSO and ARBAC, uses Hyperforce tagging to map owners to resources, checks change/incident systems to block unsafe runs, and measures experiments against existing SLOs. Game days continue for org-wide scenarios while service owners run targeted, observable experiments in development and production-like environments.
AI-written summary · about
chapters (14)
- 0:00intro
- 0:22preamble
- 0:58back to 2015
- 1:50a typical game day exercise in 2015
- 2:08evolving infrastructure and service ownership
- 3:32chaos engineering: a part of service ownership
- 4:55major scale and shift-left challenges
- 5:22challenge 1: our aws footprint
- 6:16challenge 2: multi-tenant kubernetes clusters
- 7:20challenge 3: inventory and role-based access
- 8:33challenge 4: safety, observability, and outcomes
- 9:32recommendations for a self-service chaos platform
- 11:10the ongoing role of game day exercises
- 12:49thank you
related talks
19mChaos Engineering at Scale
46mEmbracing Collaborative Chaos: Chaos Days on Large Platforms
26mService Ownership: DevOps for Salesforce
47mChaos Engineering Bottom Up
32mA Peek Inside the Enterprise Cloud at Salesforce
33mChaos Engr at Microsoft with Azure Chaos Studio
11mOrganizational Chaos & recipes for Service Ownership
39mResilience & Chaos Engineering in the Cloud