Actionable insight with derived second order monitoring data
summary
Infrastructure must emit metadata so teams derive actionable signals: use API-driven infra and CloudFormation, log function calls with decorators in serverless, and focus on business metrics and clusters not individual CPUs. Build self-service metric ingestion, alert on increases in failure rates, and apply double exponential smoothing (or Netflix hatless-style math) to avoid noise. Track 99th percentile tail latency (via t‑digest), monitor queue depth/cumulative flow to decide doubling consumers, and record event logs for every automated change.
AI-written summary · about
chapters (30)
- 0:00Start
- 0:16Infrastructure Data
- 2:16Indirect thinking
- 2:34Infrastructure as Code
- 2:45Automate all the things!
- 3:49API Based Infrastructure
- 4:10Choosing your unit of compute
- 5:18Containers for deployment vs runtime
- 7:27Observability
- 8:19Logging with Decorators
- 9:59Monitoring Containers
- 10:37Monitoring Serverless
- 11:06Actionable Metrics
- 12:37Double Exponential Smoothing
- 13:31Alert Tuning
- 17:15Self Serve is the Key
- 18:34Alert on increase of failure, not lack of success
- 19:17There are tools to help you trace your RPC calls
- 21:30You can estimate percentiles on (or streaming) datasets
- 22:37Anscombe's quartet
- 23:24Which is greater? Queues or Sliced Bread?
- 23:39Queue Depth
- 24:02Cumulative Flow Diagram
- 24:42Capacity utilization increases queues exponentially
- 25:23The price of variability
- 25:53Logs
- 27:58The Solution
- 29:02Something breaks...
- 37:04Open Port Mapping
- 38:05Questions?
related talks
19mThe Data of Infrastructures
58mObserving Applications in Production
41mMetrics Applied
36mThe road to world-class observability
42mMachine Learning In Observability Systems: From Insights To Action
39mActionable Observability
37mWhat I learned from monitoring more than 30 Machine Learning Use Cases
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale