Mastering Near-Real-Time Telemetry and Big Data: Invaluable Superpowers for Ordinary SREs
summary
Netflix Open Connect runs a purpose-built CDN on thousands of sites, peers directly with ISPs for >95% of traffic and supports 137 million subscribers; telemetry pipelines push event and verbose logs into S3/Hive and into Elasticsearch for short-term queries using Hive, Presto, Spark, Elasticsearch and internal tools plus Tableau/Cabana. Capacity tuning yields ~90 Gbps per rack unit in practice. Near-real-time alerts detect issues like broken return paths, BGP misroutes and overheating, shrinking detection to 2–3 minute detection and recovery to about 5–6 minutes.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:23How Netflix works
- 2:12Control Plane
- 2:24Content Delivery Networks
- 2:46Exceptions
- 4:18CDN
- 5:43ISP Infrastructure
- 6:19ISP Network
- 6:51Quality of Experience
- 7:38Video Quality
- 9:05Netflix Data Platform
- 10:02Data Sources and Layers
- 14:38Data Collection
- 15:25Server Temperature
- 16:31Woman
- 16:57Multidimensional Data
- 20:11Site Migration
- 21:19Latency
- 23:03Aggregating Data
- 24:05Fictional Screenplay
- 25:15Stage 2 Investigation
- 28:59Summary
related talks
31mThe Observability Data Lake, 1 year on
12mHarnessing Chaos Engineering to Build Robust Systems
42mHow Netflix Built a Real-Time Distributed Graph for Internet Scale
41mScaling Telemetry Systems with Streaming
59mAdvanced Observability Strategies for Media Workflows at Netflix
21mHandling Traffic Spikes Through Automated Prioritized Load Shedding
34mReliable OpenTelemetry at Scale: No Queue, No Problem
45mKeynote Address: Netflix: Streaming Entertainment to 200 Million Members Around the World