Making your Kubernetes-based log collection reliable & durable with Vector
summary
Vector runs as node agents that tail /var/log/pods, store checkpoints on a mounted PV, and buffer by default just 1 message so production clusters risk node disk full when outputs backpressure; fix options include increase buffer to 10,000, use disk buffer or change to drop_newest and prefer a stream topology to push logs off-node. Prometheus cardinality from file labels explodes; use expire_metric_seconds or remove the file label. Reduce etcd load by using apiserver cache-through and priority/fairness queues.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:35Kubernetes logs we collect
- 7:32Why Vector excels
- 9:18Vector architecture
- 12:09Vector topologies
- 13:55Vector in pods
- 16:46Real-world outage case no space left on device
- 19:41Prometheus explosion due to Vector metrics
- 22:34Outage: control plane
- 26:11Why Vector is great
- 26:54Q&A
related talks
28mMaking your Kubernetes-based log collection reliable & durable with
28mMaking your Kubernetes-based log collection reliable & durable with
36mTips and tricks for Kubernetes storage
25mFoster Secure Innovation with Your Kubernetes Platform
34mPanel: Improving and Managing Kubernetes at Scale
07mData Gravity and Kubernetes: Managing Large-Scale Data Ingest With Minimal Latency
27mHandling Node Churn in Karpenter: Efficient Scaling for Large EKS...
35mKubernetes, Lessons Learnt