Cross-System Interaction Failures: Don't Fail through the Cracks

on YouTube ↗40m

" summary

Empirical study of 120 cross-system incidents and a random sample of 360 issues finds roughly 30% are CSI failures that arise only when systems interact; concrete examples include Spark failing on HDFS’s legacy size = -1 and Flink retry storms overloading YARN. Two concrete defenses appear: SE, a perturbation-based tester for Kubernetes controllers that reports 46 bugs (35 confirmed, 22 fixed) across 10 controllers, and acto, an end-to-end fuzzer that finds hundreds of bugs in 50+ controllers. A formal approach proves controller correctness via eventually stable reconciliation (ESR) and verified Rust controllers.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 3:31Why failures are common
3. 5:36Reliability challenges
4. 11:15Testing interactions
5. 14:46Controllers and system state
6. 25:58Interaction between controllers and managed applications
7. 31:33Multiple controllers
8. 38:41Reliable cloud vision

" related talks

• 19mCAFault: Enhance Fault Injection Technique in Practical Distributed Systems via...Yu Jiang, Fuchen Ma, Yuanliang Chen, Yuanhang Zhou, Zhen Yan · USENIX ATC '25
• 18mEfficient Exposure of Partial Failure Bugs in Distributed Systems with Inferred...Peng Huang, Jia Pan, Haoze Wu · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)
• 31mWhat Breaks Our Systems: A Taxonomy of Black SwansLaura Nolan · SREcon19 Americas
• 14mUnderstanding and Detecting Fail-Slow Hardware Failure Bugs in Cloud SystemsYu Hua, Yongle Zhang, Gen Dong, Zhangyu Chen, Menglei Chen · USENIX ATC '25
• 30mWhat Breaks Our Systems: A Taxonomy of Black SwansLaura Nolan · LISA18
• 42mJepsen 9: A Fsyncing FeelingKyle Kingsbury · GOTO Chicago 2018
• 29mHorrors of Distributed SystemsAndrew Godwin · PyCon AU 2017
• 46mFault tolerance in Distributed Systems: A Case Study in Apache SparkImran Rashid · Scala Days Copenhagen 2017