Making your Kubernetes-based log collection reliable & durable with
summary
Vector-based agents collect pod and node logs by mounting /var/log, storing state and buffers on disk, and running as a DaemonSet with a reloader sidecar and kube-proxy metrics proxy. Real deployments across hundreds of clusters reveal three outages: disk exhaustion from retained rotated files (fixes: disk-backed buffers, drop policy, raise in-memory buffer to 10,000 messages), Prometheus metric cardinality and memory growth from a file label (fixed by removing the label rather than expire_metric_seconds), and API/etcd overload from concurrent metadata calls (mitigated with client caching and Kubernetes priority and fairness queues).
AI-written summary · about
chapters (9)
- 0:00Introduction and overview
- 1:47Types of logs in Kubernetes
- 5:18What is Vector and why use it
- 10:04Vector deployment topologies
- 12:15Deploying Vector in Kubernetes
- 16:07First real-world outage case
- 19:23Prometheus metrics explosion
- 22:37Kubernetes control plane outage
- 27:10Conclusion and key takeaways
related talks
35mMaking your Kubernetes-based log collection reliable & durable with Vector
37mWeighing a Cloud: Measuring Your Kubernetes Clusters
35mKubernetes, Lessons Learnt
35mFrom New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes
34mPanel: Improving and Managing Kubernetes at Scale
35mDeep Dive: Kubernetes (Instrumentation) SIG
42mMaking Resilient, Self-Adaptive, And Autonomous Kubernetes Clusters
19mKeynote: Monitoring Kubernetes Clusters with Prometheus