Getting Organizational Buy-In for Observability
summary
Gopuff forms an SRE team in early 2021, installs datadog across three clouds, and builds log pipelines plus metrics-from-logs to create SLOs. Engineers send two custom metrics (orders placed/orders delivered) via a function app; an example incident shows a ~21‑minute customer impact and visible recovery. A small library set called Puff Telemetry (Node/Go/Java/Python) standardizes OTLP tracing, structured logs and tags. A Puff Incident API ties Datadog, PagerDuty, Zoom and Slack to a service catalog with tiered ownership, cutting confusion and lowering MTTR.
AI-written summary · about
chapters (12)
- 0:00Start
- 2:04Topics
- 2:53gopuff
- 3:46How 2020 started
- 5:28How SRE started
- 6:40The focus for SRE
- 8:22Some data to chew on
- 15:19Why OpenTelemetry?
- 16:52PuffTelemetry
- 19:24Response
- 26:29PuffDirectory
- 29:20Lessons Learned
related talks
46mObservability in the real world: Improve systems w/iterative approach
38mAWS Observability as Code: Leveraging Datadog
21mHow Pipedrive switched its observability stack to OpenTelemetry & LGTM
33mEffective observability in microservice architectures
47mObservability - Holistic Quality across Software Systems
24mFrom firefighting to foresight: Powering innovation
27mBuilding a Scalable Monitoring System
01h 07m"Adding Observability to Distributed Systems"