Weathering the Storm: How Early Warnings Save the Farm
Talkat SREcon19 Europe/Middle East/Africa · on Thursday, 2019-10-03· at 11:45· Track 2: The Liffey A·
summary
LinkedIn builds an automated correlation engine that ingests thousands of per-endpoint sensors, call trees/traces, metrics and deployment events, precomputes relationship graphs and generates alerts using standard-deviation thresholds to correlate failures to a guilty microservice. The system returns a short ranked causal chain plus evidence, cuts MTTD/MTTR and on-call toil, and detects about 90% of major incidents by relying on precomputed relationship trees and per-endpoint alerts via standard deviation thresholds.
AI-written summary · about
chapters (9)
- 0:00Intro
- 2:12World of Microservices
- 4:36When a micro-service misbehaves?
- 7:56Relationships
- 13:01Key Goals
- 14:43Key Design Considerations
- 17:51Alert Correlation - 10000 FT Overview
- 21:53More Wins
- 29:31Next Steps
related talks
30mReducing MTTR and False Escalations: Event...
46mEvent Correlation: A Fresh Approach towards Reducing MTTR
20mSpike Detection in Alert Correlation
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
40mNext-generation Alerting and Fault Detection
16mPredictive Observability: Prevent Supply Chain Outages
13mOptimizing System Performance: O11y in Distributed Environments
44mReal Time Vulnerability Alerting by Using Principles from the United States Tsunami Warning Center