Health Checks: A Boon or a Curse?
summary
Envoy active health checks at Stripe use cached active checks via an internal cluster that serves a standardized health endpoint so backend services never see per-check traffic; health aggregates host and service state and supports local draining. At 5s intervals and 25k QPS a bad host can cause ~125k failed requests before detection; high check volume has caused hosts to burn 2+ CPU cores, hit file-descriptor limits, and respond >30s. Mitigations: shorten interval, add passive outlier detection, cache results, prune reachability, or centralize checkers; combination of active+passive scales best.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:53Conceptual Overview
- 2:57Envoy Options
- 6:42Stripe Health Checks
- 9:30Problems
- 10:28Slow Time Detection
- 12:55Issues
- 14:33Mitigations
- 18:28Solutions
- 19:48Conclusions
related talks
28mApplication Health Checks Done the Right Way
01h 02mI don't feel so well… Integrating health checks in your .NET Core solutions
21mImproving the Reliability of Cloud Foundry Applications Through Advanced Health Checks
22mWhy is it always DNS, TLS, and bad configs?
25mDon’t Catch Feelings, Catch Issues With Kuberhealthy
34mThe Doctor Is In: Using checkups to find bugs in production
31mWhy is it always DNS, TLS, and bad configs?
32mCustomer Centric Monitoring