Measuring the Reliability of Live Video Infrastructure
summary
Facebook Live defines an end-to-end reliability metric called good broadcast rate that classifies each broadcast as good or bad using real-time tracing with a unique broadcast ID across ingest, processing, delivery and CDN. Aggregation weights broadcasts by viewer share to catch high-impact failures and reveals that roughly three of ten one-hour streams will suffer issues without fixes. Operations use hourly SLOs, drilldown traces to reroute traffic (fiber-cut example), and A/B tests that once cut mobile GBR by about 5% when lowering FPS.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:50FB Live infrastructure failures
- 6:21Reliability metrics for FB Live
- 9:52Real-time data gathering
- 11:38Good Broadcast Rate (GBR)
- 13:44GBR challenges: weighting & segmentation
- 14:46GBR in daily engineering
- 16:52GBR for incident mitigation
- 17:55Summary
- 18:31Q&A on QoS, QoE & ML
related talks
10mMeasuring the live video latency at scale
01h 00mBuilding Reliable Services at NRK TV
10mMoving Live Video Quality Control from the Broadcast Facility to the Living Room
17mScaling End to End Reliability Tracking Across Large Scale, Multiplexed Products and Services
17mMetrics Driven Obsession with Viewer Experience
40mWhy Is My App SLOw? Defining Reliability in Platform Engineering
15mShit, our Site is Down! Let's build a culture of reliability
21mLive Traffic Load-Testing-Measuring and Validating Capacity at Facebook