Reliable Observability at Scale: Error Budgets for 1,000+
summary
Zendesk implements companywide error budgets for ~1,000 engineers with ~10 SREs, rolling out SLOs across teams in about four months. Implementation uses StatsD, Datadog, APM and CDN logs, avoids fragile percentiles in favor of counter-based SLIs and latency bands (histogram bins) that sum reliably over any period. Each service reports client-facing SLIs, teams keep out-of-budget playbooks, and the approach proves practical and scalable across dozens of services.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:04Agenda
- 2:26Zendesk Architecture
- 4:58History
- 6:11SLAs
- 7:35SLA Binding Targets
- 9:35Air Budgets
- 11:16Tooling
- 14:29Technical Approach
- 17:34Histograms
- 25:49Questions
related talks
19mLatency and Availability Error Budgets Done Right at Scale
30mTechniques for SLOs and Error Budgets at Scale
31mSite Reliability Engineering (SRE) and...
30mMetrics for good, not evil
27mGetting Started with SRE
38mImplementing SRE practices: SLI/SLO deep dive
17mRolling out Service Levels across a large org
29mMeasuring Reliability: What Got Us Here Won't Get Us There