Behind the Scenes: Running Critical Services @ Netflix

on YouTube ↗27m

" summary

Netflix runs the playback license system as an 18‑microservice workflow that handles license requests, content policies and device retirement, deployed active‑active across three AWS regions with weekly failover exercises. Continuous delivery happens daily (dependencies picked up each day) via Spinnaker with canaries scored by an open tool (Kadampa), end‑to‑end client tests, chaos experiments (Chaos Monkey, ChAP) and progressive rollout windows. A global outage traced to a config change on one device type leads to treating config as code and syncing gRPC and client libraries to prevent recurrence.

AI-written summary · about

" chapters (20)

1. 0:00Intro
2. 4:07Playback starts with the Manifest which contains details of the playback experience
3. 4:27The device fetches the encrypted content from the closest CDN (Content Delivery Network)
4. 4:45The device requests a license to decrypt the content and start playing
5. 5:09The licensing service supports thousands of devices with 18 microservices, 2 data stores, and a 99.999% SLA
6. 7:05Critical services are deployed in 3 AWS regions with an active-active architecture Global
7. 8:38Functional fault isolation: Divide each service into its own application while preserving a common API surface. Control traffic in the API layer
8. 10:28Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence
9. 14:05Embracing continuous deployment and automated testing in production
10. 14:38System metrics (CPU, Memory, Latency, Threads, etc..) are tested using canaries and automated analysis
11. 15:01Baseline and Canary are compared on multiple dimensions. Any deviation will affect the overall canary score
12. 15:55Service diagnosis gives instant feedback based on well-known requests in production
13. 16:40A Netflix-controlled app is redirected to an end end production cluster for comprehensive functional vetting
14. 17:53Chaos testing: Testing the unexpected. Inject failure and evaluate real time user data
15. 18:14Deploy to the non-critical cluster first, staggering each of the regions
16. 18:52Deploy to the critical cluster after gathering metrics from the non-critical cluster
17. 20:38A typical Netflix microservice is responsible for both the reliability of the server side and the implementation on the client side
18. 21:08Historically there are two main frameworks essential to implement service/client reliability: Hystrix and Ribbon
19. 22:00Protocol buffers is an agnostic declarative language to define and serialize structure data
20. 22:49GRPC allows us to define the RPC operation in protocol buffers and then auto-generates the client and server stub service NetflixService

" related talks

• 29mHow Netflix Thinks About DevOpsTejas Chopra · Developer Growth Summit 2022
• 31mHow Netflix Thinks About DevOpsDave Hahn · DevOpsDays Rockies 2016
• 19mKeynote: Making and Keeping Netflix Highly AvailableKatharina Probst · MesosCon Europe 2017
• 29mFast, Safe, and Continuous -- Picking all 3 at NetflixDave Hahn · Spinnaker Summit 2019
• 31mKeeping Your Cloud Footprint in CheckCoburn Watson · GOTO London 2015
• 58mBuilding a Culture of Continuous DeliveryAdam Jordens · Voxxed Days Banff 2018
• 44mTechniques Netflix Uses to Weather Significant Demand ShiftsJoseph Lynch · SREcon25 Americas
• 49mChaos & Intuition Engineering at NetflixCasey Rosenthal · GOTO Chicago 2016