Cultivating Production Excellence

on YouTube ↗58m

" summary

Argues for measurable production excellence: set SLIs/SLOs (example target 99.9% over 30 days), compute an error budget (1/1,000 requests → 10,000 failures/month at 10M requests) and use it to decide wakeups; ship fast and safe (builds <10 min, push hourly, 14 deploys/day) while instrumenting with OpenTelemetry and Honeycomb-style traces for live debugging. Run controlled chaos (restart one server weekly, test Kafka failover), gate risky experiments with feature flags and Terraform, and fix missing telemetry that hid a real outage.

AI-written summary · about

" chapters (15)

1. 0:00Start
2. 4:20What does uptime mean?
3. 8:22Noisy alerts. Grumpy engineers.
4. 11:00We forgot who operates systems.
5. 15:28We need Service Level Indicators
6. 18:06Set a target Service Level Objective.
7. 18:19Use a window and target percentage.
8. 29:23Observability isn't just the data.
9. 31:59Collaboration is interpersonal.
10. 32:46Lean on your team.
11. 34:47Reward curiosity and teamwork.
12. 36:54Quantify risks by frequency & impact.
13. 49:36Experiment using error budgets.
14. 52:16Test the telemetry too!
15. 56:09Buy the right tools...

" related talks

• 29mCultivating Production ExcellenceLiz Fong Jones · ITNEXT Summit 2019
• 47mCultivating Production ExcellenceLiz Fong Jones · NDC DevOps 2020
• 25mHow to detangle and understand productionLiz Fong Jones · The Lead Developer New York 2019
• 03mCultivating production excellenceLiz Fong Jones · O'Reilly Velocity Conference 2019
• 19mThink Prod Up Front: Shifting Left on Production ExcellenceShelby Spees · DevOpsDays Seattle 2021
• 39mShifting Left on Production Excellence with ObservabilityShelby Spees · DevOpsDays Boston 2021
• 29mReducing Trauma in ProductionMandi Walls, Julie Gunderson · Conf42 Site Reliability Engineering 2021
• 05mReducing Trauma in Production with SLOs and Chaos EngineeringMandi Walls · DevOpsDays NYC 2023