Journey to observability: tracking every function execution in production
summary
Describes converting IBM Cloud Ingress monitoring into true production observability by adding a Go observability context (context-based plugin model) that provides automatic instrumentation, creates a trace per execution, records histograms for every function with exemplars for trace/metric correlation, and uses OpenTelemetry for traces plus Prometheus/Grafana for metrics and dashboards. Results include a 90% reduction in incident volume and downtime, customer tickets down ~75%, critical impact events from monthly to once a year, and faster feature delivery (quarterly to monthly major releases).
AI-written summary · about
chapters (17)
- 0:00Introduction to observability journey
- 1:04Why we needed change
- 2:50Limitations of testing in production
- 4:36The observability problem cycle
- 6:03Requirements for observability solution
- 7:26Evaluating existing solutions
- 9:35Introducing observability context
- 12:08How the observability context works
- 13:54Implementation and coverage results
- 14:37Turning data into actionable insights
- 17:49New development mindset with observability
- 18:12Real-world production examples
- 20:25Impact on user experience and business
- 22:11Quantifiable business results
- 24:43Roadmap and team productivity gains
- 26:54Current status and future plans
- 27:40Q&A session
related talks
33mThe Observant Developer - OpenTelemetry from code to prod
01h 07m"Adding Observability to Distributed Systems"
13mForget Production: Observability a Non-Production Necessity
45mDeveloping Observable Applications for Production
37mThrough the looking glass: Effective observability for cloud native applications
58mObserving Applications in Production
40mContinuous Profiling, the missing piece in your observability puzzle!
31mObservability for Developers: How to Get There