The Dangers of Being Overly-Paranoid
summary
Intercom ships to production >100 times per day with 66 authors committing hundreds of times weekly, and postmortems show 70% of fixes are “add more data” while zero remove it. Logging initially produces 18 million logs in 20 minutes so teams adopt canonical log lines (nine fields: admin_id, host, release, IP, user agent, etc.) and swap Rails logger for the logging gem, which benchmarks 24.68× faster. Also: 900 dashboards, linters need caching, tests target high-risk paths.
AI-written summary · about
chapters (21)
- 0:00Intro
- 2:50AAR (After-Action Report) Meetings
- 5:35what is observability anyway?
- 11:37in the beginning there was... text-ish
- 12:08everything in one production.log file
- 12:30too much data and no structure
- 16:05a lot of data and no story
- 19:05asking questions, find pretty thing
- 20:37asking questions. benchmarking logging
- 22:12taking things for granted
- 22:59dangers of logging
- 24:59metrics. why do you care?
- 26:01metrics, metric types
- 27:30what I wanna do with it?
- 28:08metrics, what do you capture?
- 28:37metrics. operational
- 29:58metrics, engineering & analytics
- 31:21look at all these notifications now have
- 35:25why are tests here?
- 36:10story time
- 38:47development time is really important
related talks
23mSquash Production Defects Quickly - The Power of Structured Logging in Rails
28mAsk your logs
40mLog Story Short: Chopping Through Forests Of Data
40mGrowing internal tooling from the console up
39mKeeping tabs on production
36mIt's not in Production Unless it's Monitored
29mApplication Logging
44mYour App is not a Black Box