High Network Reliability & Availability in FE & BE for Scalable Training Solutions
summary
Meta runs separate FE and BE fabrics, with BE clusters built from ~18 CTS spines, ~252 RTWs per AI zone and roughly 4,000 GPUs per cluster, so a single device can affect about 1/18 of traffic. Monitoring uses dual passive/active pipelines with an on-box agent and netNORAD per-minute loss measurement; a topology-aware correlation engine cuts ~250:1 noisy detections to real root causes, integrates RMA data, and drives strict MTTR SLOs — results: fewer large-impact failures, faster repairs, and consistent capacity preservation.
AI-written summary · about
chapters (12)
- 0:00Network reliability in AI training
- 1:04Front-end fabric topology
- 2:06Backend topology & failures
- 3:51Network reliability challenges
- 6:42Predicting & mitigating failures
- 7:47Monitoring network health
- 10:11Root cause correlation
- 12:20On-box agents for triage
- 13:44Rapid repair & MTTR focus
- 14:47Reliability initiative results
- 17:34Predictive reliability future
- 19:58Key takeaways
related talks
11mBuilding Resilient Network Infrastructure
46mLessons learned from monitoring a large production network
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
21mAnatomy of @scale Machine Learning Network Operations
15mHow Our SREs Safeguard Nanosecond Performance—at Scale—in an Environment Built to Fail
21mAI Hardware Reliability at Scale
21mOperationalizing ML Training Infra at Meta Scale
19mAI/ML Networking Challenges: The Fast and the Finicky!