30 Minutes To Zero: Slashing Debug Time With WhatIsWrongWithThis?
summary
A core aggregator calling 70 downstream clients suffers 30‑minute log delays; the team builds WhatIsWrongWithThis? using non-invasive HTTP interceptors and a request‑scoped context so adding a query parameter returns timings, Hystrix state, request/response and error IDs instantly and also serializes a structured JSON stream into OpenSearch. Early issues: 10MB payloads blow JVM heaps and don’t scale, and edge failures remain invisible; the pattern cuts mean‑time‑to‑detect from tens of minutes to seconds/minutes and enables aggregate trace analytics (found a bot causing 0.5% slow checkouts).
AI-written summary · about
chapters (13)
- 0:00Welcome
- 1:04Session rules
- 2:09Cost of outages
- 3:57Debugging pain root cause
- 5:23Log gathering nightmare
- 6:45First solution attempt
- 9:17Initial limitations
- 11:06Monitoring vs observability
- 13:14Why observability matters
- 14:59Black Friday case study
- 17:27Root cause with observability
- 19:32Impact and conclusion
- 20:42Q&A
related talks
49mResponding in a Timely Manner
59mObservable Web Applications
40mCracking Microservices Performance Issues with Distributed Tracing
50mDistributed Tracing in the Wild
23mHow We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries - Bryan Boreham
39mScaling OpenSearch: Next-Generation Shard Allo...
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
46mHow to Properly Blame Things for Causing Latency