Sibylla: To Retry or Not To Retry on Deep Learning Job Failure

on YouTube ↗13m

" summary

Sibylla classifies DL job failures as deterministic or nondeterministic by training RNN models (GRU/LSTM-like) on stdout/stderr logs and updating online with an auto-labeling majority-vote scheme. Dataset includes 97 operator logs and 159 manually collected logs (text-augmented). Simulations on the Microsoft Philly trace show avoiding retries for repeated deterministic faults cuts wasted GPU hours (deterministic = 5–23% of jobs, 12–20% GPU-hours) and improves cluster utilization and JCT by up to 15%, while lowering success rate by only about 0.06%.

AI-written summary · about

" chapters (11)

1. 0:00Intro
2. 0:41Shared multi-tenant GPU clusters to
3. 1:09Problem: Job failure of DL training
4. 2:00Two types of job failures
5. 4:03Opportunity on predictive retry
6. 5:28Data source: stderr/stdout streams
7. 6:22Sibylla: Text preprocessing
8. 6:58Sibylla: Training phase
9. 9:50Can Sibylla classify failure type well
10. 10:44Can Sibylla improve cluster efficien
11. 11:42Can Sibylla maintain job success ra

" related talks

• 19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training WorkloadsFan Yang, Wencong Xiao, Shivaram Venkataraman, Myeongjae Jeon, Amar Phanishayee, Junjie Qian · USENIX ATC '19
• 19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning TrainingYuanqing Wang, Yuchen Zhang, Yu Wang, Junhao Hu, Chao Yu, Hao Lin, Zhi Yang, Zhen Guo, Si Xu, Quanlu Zhang, Chunyang Zhu, Zixiao Huang, Bo Dai, Xu Fu, Boxun Li, Guohao Dai · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)
• 21mEnhancing Runtime Reliability in LLM Training via Fine-Grained ObservabilityLei Zhang · @Scale: Networking 2025
• 31mReliable Data for Large ML Models: Principles and PracticesMary McGlohon · SREcon23 Europe/Middle East/Africa
• 17mModel-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-ServiceSameh Elnikety, Jeff Zhang, Shuayb Zarar, Atul Gupta, Siddharth Garg · HotCloud '20
• 16mUnderstanding Stragglers in Large Model Training Using What-if AnalysisXin Liu, Haibin Lin, Ziheng Jiang, Aurojit Panda, Xiang Shi, Zuquan Song, Shuguang Wang, Menghan Yu, Zhanghan Wang, Jinyang Li, Zuocheng Shi, Jinkun Lin, Sida Zhao, Chenyuan Wang, Wei Jia, Zherui Liu · OSDI '25
• 16mSLearn - A Case for Task Sampling based Learning for Cluster Job SchedulingAkshay Jajoo, Y. Charlie Hu, Xiaojun Lin, Nan Deng · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)
• 29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML PlatformMary McGlohon · SREcon21