Fault Tolerance is a Requirement Not a Feature - Adar Danait, Lilian Ernest, Citi

on YouTube ↗29m

" summary

Citi runs a 700+ service ecosystem where a request touches 5+ services and a single slow dependency under 50 RPS causes thread exhaustion and cascading latency. They instrument Hystrix to implement Hystrix circuit breaker, per-endpoint fallbacks, thread pool isolation and execution isolation threads. Demo uses a 5s injected latency, Hystrix timeout 1s and a thread pool size of 5 and shows page load collapse to ~5s then recovery: circuit trips and responses return in ~300ms then ~86–100ms after the service heals.

AI-written summary · about

" chapters (6)

1. 0:00Introduction and team intro
2. 1:05Problem with microservices dependencies
3. 7:06Circuit breaker pattern explained
4. 12:00Advanced hystrix patterns
5. 22:57Live demo with hystrix
6. 27:19Key takeaways and conclusion

" related talks

• 51mBuilding Fault Tolerant MicroservicesKristoffer Erlandsson · Jfokus 2016
• 49mThe modernization of legacy systems with microservicesHolger Kraus · JavaDay Kyiv 2015
• 40mLet it fail: resilience patterns for microservicesJoão Alves · Commit Conf 2019
• 25mFail Faster: Adding Circuit Breakers to your APIsCraig Freeman · Node+JS Interactive 2018
• 38mDistributed applications may fail. Be prepared with MicroProfile Fault ToleranceBernd Müller · EclipseCon 2023
• 42mArchitecting for Failures in MicroservicesBhakti Mehta · Devoxx US 2017
• 44m3 common pitfalls in microservice integration and how to avoid themBernd Rücker · Craft Conference 2019
• 01h 00mSystem Stable: Robust connected applications with Polly, the .NET Resilience FrameworkBryan Hogan · NDC Oslo 2019