How Meta Keeps its Large-scale Infrastructure Hardware Up and Running
summary
End-to-end remediation flow runs Machine Checker → Alertmanager → BAR auto-remediation → Cyborg reimage/firmware → repair ticket when physical touch or undiagnosed, and a data monitoring pipeline traces every failure through that flow. Reporting change uses hybrid CMCI+SMI tuning to cut CPU slow time by more than 70% without losing correctable-memory error counts. Detection streams server uptime into Scuba for near real time reboot detection and automated RCA across millions of entities, hundreds of attributes, and an ML model ranks likely repair actions for automated recommendations.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:13Hardware infrastructure importance
- 5:47Hardware remediation flow
- 9:36Tuning memory error reporting
- 11:23Near real-time reboot detection
- 13:53Automated failure root cause analysis
- 16:00Predicting repair actions
- 19:29Recap and future scaling
- 21:16Conclusion
related talks
21mAI Hardware Reliability at Scale
21mHigh Network Reliability & Availability in FE & BE for Scalable Training Solutions
35mResource Monitoring and Optimization with Metaflow
42mImproving Machine Learning Development Reliability
26mML Monitoring & Observability
22mAI Observability At Meta Scale
21mOperationalizing ML Training Infra at Meta Scale
08mAI-Powered Anomaly Detection & Self-Healing Systems