Building Resilient Monitoring at Meta
summary
Meta hardens monitoring that watches millions of hosts by building a Time Series store, petabytes of logs, tracing, and an end-to-end blackbox tester called monmon that continuously exercises alert workflows. Outage recovery reveals 10× peak load on some services, so Meta runs monthly regional failure drills with Bell Jar and Shard Manager, quarterly load‑shedding exercises, enforces tier‑zero SLOs, and uses an "always be monitoring" program plus monitoring overviewers to keep the gains.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:03How we measure resiliency
- 3:24Why are systems complex
- 6:16What do we do
- 7:44What did we do
- 8:01Identity
- 8:32Always Be Monitoring
- 8:57Building RV Teams
- 9:31Trailblazers
- 10:03What we do
- 16:18Whats next
related talks
18mMaking Deployments Safe at Meta
26mML Monitoring & Observability
23mMastering Monitoring & Alerting: Key to Seamless Performance
22mService Monitoring Manual—2018 Edition
17mRealtime Monitoring
21mHigh Network Reliability & Availability in FE & BE for Scalable Training Solutions
22mAI Observability At Meta Scale
16mMicroMon: A Monitoring Framework for Tackling Distributed Heterogeneity