Transformers in SRE Land: Evolving to Manage AI Infrastructure
summary
Ant Group describes managing thousand‑GPU training clusters where a single bad GPU causes daily multi‑hour interruptions and idle expensive hardware. Engineering adds millisecond‑resolution metrics, uses JS divergence heatmaps and frequency transforms to spot one‑GPU communication faults, defines a training time efficiency northstar metric and a proactive node benchmark protocol with five benchmark levels, and changes driver upgrades from progressive rollouts to coordinated maintenance windows to avoid repeated job restarts.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:04AI infrastructure definition
- 3:54GPU training challenges
- 7:51Microservices vs GPU training
- 8:55Reliability for training jobs
- 12:49Observability in AI infra
- 16:02Detecting degraded GPUs
- 18:53Frequency domain analysis
- 21:02Distributed tracing for GPUs
- 23:52GPU resource management
- 26:43GPU driver updates
- 28:50Node benchmarking protocol
- 32:47Key takeaways
- 35:15Q&A
related talks
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
23mStability in Large Model Training: Practices in Software and Hardware Fault Self-Healing
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
36mAccelerating AI training fleets with sched_ext
28mScaling AI Infrastructure
12mKeynote: AI Infrastructure
41mBeyond Uptime: Revolutionizing Fintech Reliability