If You Didn't Record It It Didn't Happen
summary
A Rails freelancing app called build my commits fails to show 2 of 14 GitHub commits; production logs show no errors. Instrumentation with Prometheus + Grafana, OpenTelemetry/Jaeger traces and Loki logs reveals GitHub returning 304s and a cache age of ~51.6 hours. Aggregation shows 126 users hit stale caches. Quick fix sets a 24‑hour cache threshold and forces re‑fetch; debugging drops from ~93 minutes to ~30 minutes and prevents repeated outages.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:04Missing commits investigation
- 7:00Metrics with Prometheus
- 15:33Request tracing with OpenTelemetry
- 20:36Structured logs with Loki
- 24:51Alerting to prevent issues
- 27:43Observability best practices
- 29:09Future-proofing with metrics
- 30:56Closing and demo
related talks
41mExploring the History of a 12-year-old Rails Application
23mSquash Production Defects Quickly - The Power of Structured Logging in Rails
32mOperating Rails: what about after you deploy?
25mTracking repository health at scale
43mThe Dangers of Being Overly-Paranoid
26mThe Sounds of Silence: Lessons from an 18 hour API outage
14mDatabase Performance at Scale for RoR Applications
43mAPI Optimization Tale: Monitor, Fix and Deploy (on Friday)