Keynote: From Outage To Observability: Lessons From a Kubernetes...
summary
A Kubernetes CI/CD job spikes to 2,000 pods and overloads the control plane, exposing missing correlation in logs and no tenant isolation. The remediation replaces a single Prometheus with Thanos + HA Prometheus, moves tenants into vcluster virtual clusters, adds KEDA autoscaling, ships tenant and trace IDs via Fluent Bit to OpenSearch, and experiments with Parca/Odigos for tracing and profiling. The redesigned platform isolates blast radius and recovers predictably.
AI-written summary · about
chapters (10)
- 0:00Observability crisis intro
- 1:26Kubernetes outage night
- 3:12Root causes of outage
- 4:13Thanos and vcluster monitoring
- 5:17KEDA autoscaling and isolation
- 6:00Logging tracing profiling
- 7:01AI agents observability challenges
- 9:10Fixes and success metrics
- 10:33Building resilient systems
- 10:54Closing remarks
related talks
05mKeynote: Observability in the Next Decade, Beyond Reactive Response
35mKubernetes, Lessons Learnt
20mKeynote: Finding the Joy in Chaos Engineering
42mMastering Kubernetes: Challenges, Strategies, and Best Practices
20mKeynote
19mKeynote: Monitoring Kubernetes Clusters with Prometheus
46mAdvanced Kubernetes: Lesson Learned From Building..
09mKeynote: Scaling Observability Pipelines in the Enterprise