Machine Learning In Observability Systems: From Insights To Action
summary
Nike runs an observability platform that ingests 2 million alerts a day and reduces noise via an event-to-incident funnel that compresses signals to events-to-incidents 15:1. Examples include using Prometheus and OpenTelemetry for time-series anomaly detection, Honeycomb-style heatmaps for outliers, ClickHouse/Splunk for query translation via LLMs (achieving ~85% useful answers), prescaling recommendations for shoe drops, deleting 12,000 stale dashboards, and preventing outages that previously cost about $30 million.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:14Observability and business value
- 4:40Nike's observability investment
- 8:17Telemetry signals and OpenTelemetry
- 10:25Data volume challenges
- 13:37ML use cases in observability
- 16:49Reducing alert noise
- 20:04Predictive maintenance models
- 22:12Resource allocation for traffic
- 26:03LLMs for query simplification
- 29:42Real-time insights and dashboards
- 31:52AI-driven observability culture
- 33:15From data to business value
- 35:01AI impact on incidents
- 37:10Start with open-source
- 38:16Q&A on filtering effectiveness
- 40:22Q&A on reinforcement learning
related talks
33mBeyond Monitoring: The Rise of Observability Platform
01h 00mWhy observability requires the marriage of AI, metrics, and logs
29mObservability; a gentle introduction
26mML Monitoring & Observability
37mWhat I learned from monitoring more than 30 Machine Learning Use Cases
28mScaling Observability for Real-Time Personalization Engines
21mFrom Black Box to Glass Box: AI-Driven Observability for Modern APIs
18mAI Observability: How To Fix Issues With Your ML Model