Squash Production Defects Quickly - The Power of Structured Logging in Rails
summary
Friday on-call incident where password reset emails delay because analytics jobs queue behind a scraper hitting profile show is diagnosed in five minutes by adding structured logging with Rails 8, using semantic_logger, Faraday middleware for API requests, and AppSignal output; logs follow OpenTelemetry semantic names and show job class + duration so the culprit IP is blocked in Cloudflare and queue latency drops. Follow a five-step SOS cycle (question, instrument, deploy, query, improve). Outcome: 98% downtime reduction, 83% fewer 500s, bugs fixed ~20× faster.
AI-written summary · about
chapters (19)
- 0:00Structured logging in Rails
- 3:04Production visibility issues
- 4:31SOS observability process
- 7:04Choosing log data
- 8:10Structured vs plain text logs
- 10:20Logging library selection
- 11:41OpenTelemetry standards
- 13:27Adding log attributes
- 14:32API request logging
- 14:52Sending logs to tools
- 15:36Log analysis with graphs
- 16:39Improving observability
- 17:21Observability culture
- 17:43Resolving production bug
- 20:29Fix impact measurement
- 20:50Observability superpower
- 21:12Unexpected logging benefits
- 21:5718-month results
- 22:19Resources and closing
related talks
28mAsk your logs
35m"10 performance sins of Rails applications"
43mThe Dangers of Being Overly-Paranoid
26mThe Sounds of Silence: Lessons from an 18 hour API outage
43mAPI Optimization Tale: Monitor, Fix and Deploy (on Friday)
41mScalable Observability for Rails Applications
32mRails Sustainable Productivity
35mTales from the Crypt