Unveiling Resilience: iFood's Jenga & Sherlock Holmes Practices in Chaos Engineering
summary
iFood uses Litmus Chaos across 17 production and 13 non‑prod clusters with ~300 active users and ~200 experiment runs per month, after a breaking Litmus v2→v3 migration. Two resilience practices run targeted game days: Jenga events map the end‑to‑end order flow, run 17 times in 2024, test 36 services, and uncover 160 bugs with 138 fixed; Sherlock Holmes runs stealth experiments (two events so far) that expose alerting gaps and circuit breakers that don’t open under injected network faults.
AI-written summary · about
chapters (16)
- 0:00iFood resilience journey
- 1:11iFood scale in 2024
- 3:42Resilience challenges before chaos
- 5:25COVID-19 outage and chaos birth
- 7:14Chaos engineering adoption challenges
- 9:01Team training and Litmus improvement
- 11:11Litmus adoption and experiment types
- 15:26Jenga practice for critical flows
- 16:31How Jenga works
- 18:16Mapping critical flows and engaging teams
- 20:03Jenga outcomes and 2024 fixes
- 20:45Sherlock Holmes deep dive
- 21:50Uncovering hidden weaknesses
- 23:15Sherlock Holmes results and insights
- 24:38Future resilience plans
- 26:24Q&A on chaos challenges
related talks
30mBringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering
26mBreaking Systems for Resilience
39mResilience & Chaos Engineering in the Cloud
39mPracticing Cloud-Native Resliency Is a Challenge: Decoding It with Chaos...
29mEngineering Chaos for Resilient Applications: A Gameday Guide
33mCase Study : Improving Resilience of Applications in Telco Environme...
26mFrom experimentation to continuous verification
18mChaos Experiments under the lens of AIOps