Madaari: Ordering for the Monkeys
summary
Claims that ordering LDFI finds the most likely failures first and prunes huge fault spaces using actual traces and SAT/ILP solving. eBay builds lineage from distributed traces or log‑stitching, drives a homegrown injector Mowgli, and runs experiments that drop from millions of combos to just 17 (one run) or 7 (another), discovers six real bugs, and prunes large subgraphs when key nodes survive. Results show ordered faults beat uniform random injection and integrate with existing observability.
AI-written summary · about
chapters (17)
- 0:00Intro
- 2:19The Problem: Testing Distributed Systems
- 3:30Chaos Engineering: A Possible Solution
- 4:32Chaos Engineering : A Brief Primer
- 6:17Lineage Driven Fault Injection aka LDFI
- 7:32LDFI : Building Blocks
- 9:00Encoding the Lineage
- 10:16Ordering For Faults: Injecting Faults That Matter
- 11:38Ordering Faults: Injecting Faults That Matter
- 15:24Traces = Explanations
- 16:25Fault Injection Tool
- 17:35Interaction Replay
- 18:29LDFI in the Enterprise
- 19:25Results: Finding Bugs
- 23:23Madaari: The Road Ahead
- 24:45LDFI : The Road Ahead
- 26:58Madaari: Key Takeaways
related talks
41mResillient Microservices without the Chaos
57mLearning to Love your Non-functionals
40mChaos Engineering at Scale
34mChaos Debugging: Finding and Fixing Microservice Weak Points
29mUnveiling Resilience: iFood's Jenga & Sherlock Holmes Practices in Chaos Engineering
46mTroubleshooting Distributed Workloads ...
46mPintrace: A Distributed Tracing Pipeline
30mMeasuring Microservices