Observability for Emerging Infra: What Got You Here Won't Get You There
summary
Argues that LAMP-era metrics fail for modern, chaotic systems and claims observability needs raw, contextual events not aggregated numbers; says average API request produces ~50 events (Facebook hits hundreds) and Instagram samples as low as 0.1% to manage volume. Shows how distributed tracing helps once a path is known but advocates for high-cardinality events and even building a custom storage engine to make every request searchable, plus canary rollouts (1% → 5% → 10%) and richer sampling to debug rare failures.
AI-written summary · about
chapters (37)
- 0:00Introduction
- 1:33Monitoring is dead
- 3:10I hate monitoring
- 3:33Observability became contested
- 3:48Complexity is exploding
- 4:41What is observability
- 5:39Why we drink
- 6:41Observability is about
- 7:34Complexity is rising
- 8:48How to measure complexity
- 10:07Lampstack vs Parse
- 11:37Example problems
- 12:47Best practices
- 12:59Examples
- 13:59PTSD
- 15:50Unknown Unknowns
- 17:18Symptoms
- 19:22Instrumentation
- 20:06Metrics
- 20:54Metrics vs Events
- 23:34Cardinality
- 25:04Platforms
- 25:53Structured Data
- 27:10Dashboards
- 27:24Why I Hate Dashboards
- 27:34Every Dashboard is an Artifact of a Past Failure
- 29:14Dashboards Must Die Once I Am Queen
- 29:25Every Single Needle Should Be Findable
- 30:02Right Time Aggregation
- 30:27Black Swans
- 32:26Test in Production
- 34:41Generalist Software Engineers
- 35:57Excessive Min
- 36:10WorkLife Balance
- 36:34Watching It Run
- 37:17Distributed Systems
- 37:41What You Win
related talks
43mObservability for Emerging Infra: What Got You Here Won't Get You There
44mObservability for Emerging Infra: What Got You Here Won’t Get You There
46mObservability for Emerging Infra: What Got You Here Won't Get You There
38mOff the Beaten Path: Moving Observability Focus from Your Service
33mKeynote: Observability is due for a version change; Are you ready for it?
52mObservability: it's not just an ops thing
29mObservability; a gentle introduction
38mThe Murphy's Laws of Observability