Building Reliable Services at NRK TV
summary
NRK TV serves ~500k users daily and defines reliability with concrete SLOs (example: 99.95% uptime → 4 minutes downtime/week) and per-minute SLI measurements (error rate, 99th‑percentile latency). Reliability work uses load and fault injection experiments with k6, Polly and Simi controlled by Unleash and observed in Application Insights. A concrete fix: replace on‑request blob fetch with a periodic hosted fetch plus hard‑coded fallback, which makes injected 60s latency irrelevant and prevents cache‑stampede failures.
AI-written summary · about
chapters (24)
- 0:00Start
- 1:26Agenda
- 2:39What Is the Service
- 7:02What Does It Mean To Be Robust and What Did It Mean To Be Resilient
- 7:53Resiliency
- 15:42How Complex Systems Fail
- 17:12Site Reliability Engineering
- 17:30Service Level Indicators
- 17:55Latency
- 27:37Application Insights
- 29:22Notifications
- 30:28Architecting for Reliability
- 33:29Robustness
- 35:23Fault Injection Library
- 36:06Pure Load Test
- 37:17Load Tests with Introduced Faults
- 41:17Caching
- 42:23Works Board's Time Expansion Algorithm
- 42:52Daily News
- 46:17Why Is this Api Important
- 47:15Load Test
- 49:00Latency Config
- 50:51Original Design
- 59:17Bionic Repo
related talks
15mShit, our Site is Down! Let's build a culture of reliability
37mRelia...bility?
27mContinuous Reliability. How?
40mWhy Is My App SLOw? Defining Reliability in Platform Engineering
49mScaling Reliability: So You Want to Add a 9
38mImplementing SRE practices: SLI/SLO deep dive
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
48mBuild Trust with Your Users Workshop