Automatic Generation of Runtime Checkers for Production Distributed Systems
summary
Three automated techniques generate runtime checkers: Panorama statically instruments requester code so observers report cross-component errors and detects 15 real-world gray failures across HDFS and ZooKeeper in under seven seconds; OmegaGen uses program reduction to build lightweight mimic watchdogs that catch 20 of 22 real bugs with median detection 4.2s and low isolation overhead; Oskeeper infers 1,600 semantic rules from regression tests and finds six of seven new semantic violations at ~1.27% runtime overhead.
AI-written summary · about
chapters (39)
- 0:00Intro
- 0:57Runtime checker (aka. detector/monitor)
- 1:30Importance of runtime checker
- 2:14Current checking practice
- 2:56Complex internals of modern software
- 3:20Common to exhibit gray failures
- 3:48A real-world gray failure
- 4:57Failure root cause
- 5:36Ideal runtime checkers
- 7:48A new approach
- 8:45Panorama: capture in-situ observability
- 9:25Convert a program into in-situ observer
- 10:13Identify observation boundary and identities
- 10:49Extract evidence
- 11:16Example of analysis
- 12:28Detecting real-world gray failures
- 12:54Timeline of detecting failure case f1
- 13:28Latency overhead to observers
- 15:42Program reduction approach
- 16:00Why doing reduction?
- 17:22identify long-running regions
- 18:27select checking target candidates
- 19:40reduce long-running methods
- 21:03encapsulate checkers
- 21:49insert watchdog hooks
- 22:18Prevent side effects
- 23:04Watchdog generation
- 23:28Failure detection evaluation setup
- 23:57Detecting real-world failures
- 24:50Silent semantic violations
- 25:42Real-world failure study
- 26:38Oathkeeper: detect silent semantic violation
- 27:31How to express semantics?
- 28:24Oathkeeper workflow
- 29:06Emitting semantic event traces
- 29:48General semantic rule templates
- 32:19Extracted semantic rules
- 33:17Runtime overhead
- 33:43Conclusions
related talks
19mUnderstanding, Detecting and Localizing Partial Failures in Large System Software
20mZero-Configuration Runtime Software Component Detection
37mAutomatically Detect the Top Performance & Scalability Issues in Distributed
43mSelf-Aware Applications: Automatic Production Monitoring
51mSecurity Observability 101: Thinking Inside the Box!
46mDiagnosing Production Outages — Why Applications Stop Working
46mObservability in the real world: Improve systems w/iterative approach
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise