~/talk

Evolution of Aegis: Fault Diagnosis for AI Model Training Service in Production

on YouTube 16m
Talkat NSDI '25 · 2025-04-28..04-30· [machine-learning, observability]

speakers

name:Jianbo Dong
links:none on file
name:Kun Qian
links:none on file
name:Pengcheng Zhang
links:none on file
name:Zhilong Zheng
links:none on file
name:Liang Chen
links:none on file
name:Fei Feng
links:none on file
name:Yichi Xu
links:none on file
name:Yikai Zhu
links:none on file
name:Gang Lu
links:none on file
name:Xue Li
links:none on file
name:Zhihui Ren
links:none on file
name:Zhicheng Wang
links:none on file
name:Bin Luo
links:none on file
name:Peng Zhang
links:none on file
name:Yang Liu
linkedin:yang-liu-796211b1
name:Yanqing Chen
links:none on file
name:Yu Guan
links:none on file
name:Weicheng Wang
links:none on file
name:Chaojie Yang
links:none on file
name:Yang Zhang
linkedin:yzhang
name:Man Yuan
links:none on file
name:Hanyu Zhao
links:none on file
name:Yong Li
links:none on file
name:Zihan Zhao
links:none on file
name:Shan Li
links:none on file
name:Xianlong Zeng
links:none on file
name:Zhiping Yao
links:none on file
name:Binzhang Fu
links:none on file
name:Ennan Zhai
linkedin:ennan-zhai-26818b17
name:Wei Lin
links:none on file
name:Chao Wang
links:none on file
name:Dennis Cai
linkedin:dennis-cai-45699a2b

resources

video:https://youtu.be/c8CvJDCDqbA
slides:none on file

summary

Aegis is a production fault diagnosis system for AI model training that evolves from enhanced legacy tools to a customized collective communication library (CCL) for runtime localization without changing customer code. It adds a pre-delivery validation step and a degradation detector using 20+ metrics and z‑score outlier analysis. Deployed for a year at Alibaba Cloud, Aegis claims >97% reduction in idle diagnosis time, 84% fewer restarts and 71% less performance degradation.

AI-written summary · about

chapters (9)

  1. 0:00Aegis introduction
  2. 1:42Large-scale AI training challenges
  3. 3:51Existing diagnosis limits
  4. 5:34Aegis Phase one
  5. 8:01Procedure-aware runtime diagnosis
  6. 10:30Performance degradation system
  7. 12:38Pre-delivery checks
  8. 13:44Aegis production results
  9. 15:31Conclusion and future

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.