You can't test everything, but you should monitor it
summary
Warehouse downloads slow from ~12 MB/s to under 1 MB/s causing a 6‑minute job to stretch to an hour; the regression exists since 2020 — two years unnoticed — and breaks fulfillment during high season when booths turn around in ~10–15 minutes. Team exports Slack logs, aggregates daily JSON, and then pushes historical and live metrics into OpenSearch via filebeat ingestion, builds dashboards with median/max/min download rates, and creates monitors and Slack alerts so problems get noticed within hours — implemented end to end in one day.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:54The photo booth cycle
- 3:40The photo booth problem
- 6:13What could it be
- 8:17Lets start
- 8:58What happened
- 9:42Preventing a test
- 11:56Monitoring alerts
- 15:18Whats next
- 17:12Feedback notes
related talks
41mScaling Observability: From Chaos to Clarity with Open Source Tools
39mSPACE and Beyond - Measuring Developer Productivity
34mMetrics driven development
30mReliable Data Processing with Minimal Toil
12mUnlocking the Secrets of Client Data
42mMake your logs work for you
37mCloud warehouse monitoring ( Snowflake case study )
52mReal-time-first metrics