Self-Service Chaos Engineering: Fitting Gremlin into Grubhub's DevOps Culture
summary
Gremlin rolls out across pre-prod with LDAP auth and broad engineer access, hitting 2,000 clients in a week, and integrates into existing workflows via a custom Spinnaker pipeline that triggers Gremlin attacks alongside Gatling load tests kicked off by Jenkins. Concrete wins: setting host clocks surfaces certificate failures, injected latency reveals hidden timeout locations, and blackholing traffic exposes Elasticsearch request-queue floods; teams use attacks to validate SLOs and fixes. Roadmap: CI integration, gated production experiments, role-scoped permissions and periodic game days.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 2:52Why self-service culture
- 5:41Why chaos engineering
- 8:50Technical rollout process
- 10:35Driving adoption and education
- 13:43Results and benefits
- 15:47Future plans and next steps
- 18:16Q&A on self-service model
related talks
46mPutting Gremlins in the Pipes: How to Automate Chaos Engineering
29mThe Road to Reliability
45mGameDays: Practice Thoughtful Chaos Engineering
46mDesigning a System for Chaos
27mImplementing Chaos Engr. in a Risk Averse Culture
57mLearning to Love your Non-functionals
18mFostering Chaos Engineering Culture
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure