The House is on fire: mysteries of on-call K8s engineering
summary
KUspaces is pitched as namespace as a service built on vcluster and an Envoy Gateway and aims to hide cluster ops while exposing the Kubernetes API. Real incidents include a 5,000 node API limit causing AKS control plane latency, massive Prometheus installs hitting OOM so teams move to Prometheus per-app, and an S3 Glacier restore that turns into an eight hour outage. Practical mitigations are hard pod caps, separate upgrade channels, better backups, and tools like Falco BPF, k9s and crew plugins.
AI-written summary · about
chapters (8)
- 0:00Introduction and Opening Remarks
- 0:27Speaker Background and Credentials
- 1:16Introduction to KubeSpaces
- 2:41Failures in Cloud Native Environments
- 4:34Interactive Session: Sharing Failure Stories
- 14:07Mitigation Strategies and Best Practices
- 17:54Tools and Recommendations
- 20:14Philosophy and Closing Thoughts
related talks
30mK8s Under Fire: Igniting Performance & Burning Through Bottlenecks with Kube-Burner
27mA Safari of Kubernetes and its Natural Habitat
37mkubernetes on AWS gone wild
45mKubernetes Operability Tooling
11mK8s Resources Ensuring Performance and Optimization
35mKubernetes journey
35m10 Weird Ways to Blow Up Your Kubernetes
33mKubernetes Security: Horror Stories to Help Secure Your Cluster